2024-09-11T14:31:40.688 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T14:31:40.702 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:31:40.778 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373 branch: main description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_cls_tests} email: null first_in_suite: false job_id: '7901373' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a workunit: branch: wip-matanb-crimson-seastore-logs env: CRIMSON_COMPAT: '1' sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5813 sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-seastore-logs suite_path: /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa suite_relpath: qa suite_repo: https://github.com/matan-b/ceph.git suite_sha1: dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvIoVkVwDQBXSOxXJnqBzu3AVe9GMpZeH4YI7axkJA9vponZ7MFXR1XCxDjBDxn7sRmFzV6sdR3nPOC+ruIx2o= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - cls/test_cls_rbd.sh - cls/test_cls_lock.sh - cls/test_cls_journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-11_13:56:22 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T14:31:40.778 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa; will attempt to use it 2024-09-11T14:31:40.779 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa/tasks 2024-09-11T14:31:40.779 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T14:31:40.780 INFO:teuthology.task.internal:Checking packages... 2024-09-11T14:31:40.803 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '0d61375b8ab53be4c233f5b93e463edfdff9316a' 2024-09-11T14:31:40.803 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T14:31:40.803 INFO:teuthology.packaging:ref: None 2024-09-11T14:31:40.803 INFO:teuthology.packaging:tag: None 2024-09-11T14:31:40.803 INFO:teuthology.packaging:branch: main 2024-09-11T14:31:40.803 INFO:teuthology.packaging:sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:31:40.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-11T14:31:41.012 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4872.g0d61375b 2024-09-11T14:31:41.014 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T14:31:41.020 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T14:31:41.020 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T14:31:41.153 INFO:teuthology.task.internal:Saving configuration 2024-09-11T14:31:41.164 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T14:31:41.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T14:31:41.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 14:27:15.162282', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvIoVkVwDQBXSOxXJnqBzu3AVe9GMpZeH4YI7axkJA9vponZ7MFXR1XCxDjBDxn7sRmFzV6sdR3nPOC+ruIx2o='} 2024-09-11T14:31:41.225 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T14:31:41.230 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-11T14:31:41.231 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T14:31:41.294 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f822a2afac0>, signals=[15]) 2024-09-11T14:31:41.295 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T14:31:41.300 INFO:teuthology.task.internal:Opening connections... 2024-09-11T14:31:41.300 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:31:41.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:31:41.391 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T14:31:41.397 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-09-11T14:31:41.471 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-09-11T14:31:41.471 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-09-11T14:31:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-09-11T14:31:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-09-11T14:31:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-09-11T14:31:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T14:31:41.530 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T14:31:41.531 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-09-11T14:31:41.548 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T14:31:41.555 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T14:31:41.586 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T14:31:41.586 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-09-11T14:31:41.603 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T14:31:41.616 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T14:31:41.616 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-09-11T14:31:41.671 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T14:31:41.703 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T14:31:41.734 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-09-11T14:31:41.734 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-09-11T14:31:41.734 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T14:31:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:31:41.752 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-11T14:31:41.810 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-503.el9.x86_64 2024-09-11T14:31:41.810 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-503.el9.x86_64 2024-09-11T14:31:41.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-09-11T14:31:44.189 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-11T14:31:49.832 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-11T14:31:56.274 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-11T14:31:57.808 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-11T14:31:58.570 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-11T14:32:04.301 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-11T14:32:06.392 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T14:32:06.392 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T14:32:06.484 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T14:32:06.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 79 M 2024-09-11T14:32:06.494 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 126 M 2024-09-11T14:32:06.495 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:32:06.803 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 79 kB/s | 23 kB 00:00 2024-09-11T14:32:12.136 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.8 MB/s | 16 MB 00:05 2024-09-11T14:32:37.443 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 954 kB/s | 29 MB 00:30 2024-09-11T14:32:44.801 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 924 kB/s | 35 MB 00:38 2024-09-11T14:32:44.801 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:32:44.801 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.1 MB/s | 79 MB 00:38 2024-09-11T14:32:45.510 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:32:45.902 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:32:45.902 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:32:46.910 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:32:46.910 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:32:49.499 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:32:50.187 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:32:50.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:32:53.077 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:32:53.207 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:32:58.338 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:32:58.382 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:03.525 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:42.326 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:42.367 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T14:33:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T14:33:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T14:33:43.553 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T14:33:43.553 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:33:43.553 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:33:43.554 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:33:43.902 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:33:43.990 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:33:43.991 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:33:43.991 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T14:33:43.991 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T14:33:43.991 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T14:33:43.991 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T14:33:43.991 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T14:33:43.991 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T14:33:44.050 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T14:33:44.050 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T14:33:44.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-09-11T14:33:44.081 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-09-11T14:33:44.081 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-09-11T14:33:44.081 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2024-09-11T14:33:44.081 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 82.el9 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8274670 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-09-11T14:33:44.082 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:33:44.083 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T14:33:44.084 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T14:33:44.084 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-505.el9.x86_64 2024-09-11T14:33:44.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T14:33:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-09-11T14:33:46.100 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T14:33:46.128 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-09-11T14:33:46.131 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-11T14:33:46.145 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.Sc1qGNhIZW 2024-09-11T14:33:46.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Sc1qGNhIZW 2024-09-11T14:33:46.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.Sc1qGNhIZW 2024-09-11T14:33:46.363 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.Sc1qGNhIZW is 6KB 2024-09-11T14:33:46.411 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.Sc1qGNhIZW 2024-09-11T14:33:46.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T14:33:46.496 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T14:33:46.496 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T14:33:46.496 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T14:33:46.496 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T14:33:46.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T14:33:46.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-09-11T14:34:16.688 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T14:34:16.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:34:16.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:34:35.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-09-11T14:34:44.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:34:44.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:35:44.267 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T14:35:56.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:35:56.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:35:56.556 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-11T14:35:56.943 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:35:56.943 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T14:35:56.943 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-11T14:35:56.961 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-505.el9.x86_64 2024-09-11T14:35:56.961 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T14:35:56.961 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T14:35:56.961 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T14:35:57.962 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T14:35:57.962 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T14:35:58.053 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-09-11T14:35:58.074 DEBUG:teuthology.parallel:result is None 2024-09-11T14:35:58.074 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T14:35:58.081 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T14:35:58.081 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T14:35:58.100 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T14:35:58.125 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T14:35:58.159 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T14:35:58.159 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T14:35:58.198 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T14:35:58.204 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T14:35:58.204 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T14:35:58.280 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:35:58.292 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T14:35:58.294 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T14:35:58.300 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T14:35:58.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T14:35:58.329 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T14:35:58.396 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T14:35:58.397 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T14:35:58.414 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:35:58.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T14:35:58.551 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:35:58.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T14:35:58.680 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:35:58.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T14:35:58.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-09-11T14:35:58.846 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:35:59.104 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T14:35:59.112 INFO:teuthology.task.internal:Starting timer... 2024-09-11T14:35:59.112 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T14:35:59.186 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T14:35:59.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-09-11T14:35:59.285 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-09-11T14:35:59.287 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T14:35:59.287 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-09-11T14:35:59.336 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-09-11T14:35:59.336 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive'} 2024-09-11T14:35:59.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:35:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:35:59.363 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T14:35:59.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-09-11T14:35:59.433 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T14:35:59.442 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T14:35:59.445 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T14:35:59.556 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T14:35:59.567 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T14:35:59.568 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T14:39:43.877 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2024-09-11T14:39:43.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:39:43.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T14:39:43.962 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-11T14:39:44.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-11T14:39:44.041 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T14:39:44.052 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T14:39:44.052 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T14:39:44.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:39:44.111 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T14:39:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T14:39:44.152 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-09-11T14:39:44.164 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-09-11T14:39:44.179 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T14:39:44.194 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T14:39:44.247 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:39:44.251 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:39:44.251 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-11T14:39:44.251 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:39:44.252 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:39:44.252 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:39:44.252 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:39:44.252 INFO:teuthology.run_tasks:Running task install... 2024-09-11T14:39:44.260 DEBUG:teuthology.task.install:project ceph 2024-09-11T14:39:44.260 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}} 2024-09-11T14:39:44.260 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'} 2024-09-11T14:39:44.260 INFO:teuthology.task.install:Using flavor: crimson 2024-09-11T14:39:44.266 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T14:39:44.266 INFO:teuthology.task.install:extra packages: [] 2024-09-11T14:39:44.266 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'tag': None, 'wait_for_package': False} 2024-09-11T14:39:44.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:39:44.407 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/ 2024-09-11T14:39:44.408 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4872.g0d61375b 2024-09-11T14:39:44.518 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/0d61375b8ab53be4c233f5b93e463edfdff9316a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T14:39:44.518 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:39:44.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T14:39:44.546 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-11T14:39:44.546 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0d61375b8ab53be4c233f5b93e463edfdff9316a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T14:39:44.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T14:39:44.689 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T14:39:44.754 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-09-11T14:39:44.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-09-11T14:39:44.995 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-09-11T14:39:45.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-09-11T14:39:45.588 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 330 kB/s | 89 kB 00:00 2024-09-11T14:39:45.881 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-11T14:39:46.156 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 8.8 kB/s | 2.1 kB 00:00 2024-09-11T14:39:47.415 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.2 MB 00:01 2024-09-11T14:39:49.183 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-11T14:39:54.134 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.5 MB 00:00 2024-09-11T14:39:55.593 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 113 kB/s | 19 kB 00:00 2024-09-11T14:39:56.697 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-11T14:40:02.395 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-11T14:40:04.867 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 10 M 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.8 M 2024-09-11T14:40:04.880 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.1 M 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 20 M 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 25 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 808 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-11T14:40:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 558 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.4 M 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 45 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 143 k 2024-09-11T14:40:04.882 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 159 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 316 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 303 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 99 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-09-11T14:40:04.883 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout:Install 31 Packages 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-09-11T14:40:04.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:04.885 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 60 M 2024-09-11T14:40:04.885 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:40:05.521 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-base-19.3.0-4872.g0d61375b.el9.x86 12 MB/s | 6.1 MB 00:00 2024-09-11T14:40:05.580 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-selinux-19.3.0-4872.g0d61375b.el9. 430 kB/s | 25 kB 00:00 2024-09-11T14:40:05.680 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): libcephfs2-19.3.0-4872.g0d61375b.el9.x8 7.9 MB/s | 808 kB 00:00 2024-09-11T14:40:05.772 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): libradosstriper1-19.3.0-4872.g0d61375b. 6.0 MB/s | 558 kB 00:00 2024-09-11T14:40:06.064 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): ceph-radosgw-19.3.0-4872.g0d61375b.el9. 10 MB/s | 10 MB 00:01 2024-09-11T14:40:06.281 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): ceph-common-19.3.0-4872.g0d61375b.el9.x 16 MB/s | 20 MB 00:01 2024-09-11T14:40:06.298 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-ceph-argparse-19.3.0-4872.g0d61 193 kB/s | 45 kB 00:00 2024-09-11T14:40:06.340 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-ceph-common-19.3.0-4872.g0d6137 2.4 MB/s | 143 kB 00:00 2024-09-11T14:40:06.415 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): librgw2-19.3.0-4872.g0d61375b.el9.x86_6 8.4 MB/s | 5.4 MB 00:00 2024-09-11T14:40:06.440 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-cephfs-19.3.0-4872.g0d61375b.e 1.1 MB/s | 159 kB 00:00 2024-09-11T14:40:06.466 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-rados-19.3.0-4872.g0d61375b.el 2.5 MB/s | 316 kB 00:00 2024-09-11T14:40:06.499 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-rgw-19.3.0-4872.g0d61375b.el9. 1.7 MB/s | 99 kB 00:00 2024-09-11T14:40:06.525 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): python3-rbd-19.3.0-4872.g0d61375b.el9. 3.5 MB/s | 303 kB 00:00 2024-09-11T14:40:06.683 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 493 kB/s | 107 kB 00:00 2024-09-11T14:40:06.742 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 704 kB/s | 171 kB 00:00 2024-09-11T14:40:06.767 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 662 kB/s | 160 kB 00:00 2024-09-11T14:40:06.792 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 415 kB/s | 45 kB 00:00 2024-09-11T14:40:06.843 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-11T14:40:06.868 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-11T14:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-09-11T14:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-11T14:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 390 kB/s | 36 kB 00:00 2024-09-11T14:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-11T14:40:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-11T14:40:07.070 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-11T14:40:07.095 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-09-11T14:40:07.120 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-11T14:40:07.145 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.4 MB/s | 191 kB 00:00 2024-09-11T14:40:07.188 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-11T14:40:07.238 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-11T14:40:07.313 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-11T14:40:07.422 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): librados2-19.3.0-4872.g0d61375b.el9.x8 15 MB/s | 3.5 MB 00:00 2024-09-11T14:40:07.505 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): librbd1-19.3.0-4872.g0d61375b.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-11T14:40:07.510 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:07.510 INFO:teuthology.orchestra.run.smithi050.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-11T14:40:07.629 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:40:07.670 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:40:07.670 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:40:08.340 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:40:08.340 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:40:09.907 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:40:10.404 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-11T14:40:10.524 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:40:10.608 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:40:10.709 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:40:10.734 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:40:10.780 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:40:10.842 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-11T14:40:10.880 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-11T14:40:10.905 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-11T14:40:10.928 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:40:10.961 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-11T14:40:11.157 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:40:11.187 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-11T14:40:11.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-11T14:40:11.266 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 11/35 2024-09-11T14:40:11.295 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:40:11.385 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-11T14:40:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:40:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 14/35 2024-09-11T14:40:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-11T14:40:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-11T14:40:11.666 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-11T14:40:11.708 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-11T14:40:11.749 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 19/35 2024-09-11T14:40:11.778 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-11T14:40:11.815 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-11T14:40:12.192 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-11T14:40:12.249 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:40:12.342 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 23/35 2024-09-11T14:40:12.370 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 24/35 2024-09-11T14:40:12.415 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-11T14:40:13.123 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-11T14:40:13.151 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:40:13.223 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 27/35 2024-09-11T14:40:13.304 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 28/35 2024-09-11T14:40:13.337 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 29/35 2024-09-11T14:40:15.893 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:40:15.925 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:40:16.657 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 30/35 2024-09-11T14:40:16.696 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:40:16.811 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 31/35 2024-09-11T14:40:16.811 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T14:40:16.811 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T14:40:16.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:16.847 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-09-11T14:40:27.563 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:28.699 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 33/35 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T14:40:28.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:28.728 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:40:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-11T14:40:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 1/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 3/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 4/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 5/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_6 6/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 7/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9. 8/35 2024-09-11T14:40:30.211 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x8 9/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 10/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 11/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 12/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 13/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-11T14:40:30.212 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-11T14:40:30.213 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-11T14:40:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-11T14:40:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-11T14:40:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 32/35 2024-09-11T14:40:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-11T14:40:30.214 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 34/35 2024-09-11T14:40:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T14:40:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:31.153 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:40:31.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-09-11T14:40:32.109 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:30 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:40:32.247 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:40:32.248 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:32.248 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:40:32.248 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 85 M 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 175 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-11T14:40:32.249 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:Install 9 Packages 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 87 M 2024-09-11T14:40:32.250 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 797 M 2024-09-11T14:40:32.251 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:40:32.990 INFO:teuthology.orchestra.run.smithi050.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 480 kB/s | 94 kB 00:00 2024-09-11T14:40:33.015 INFO:teuthology.orchestra.run.smithi050.stdout:(2/9): libcephsqlite-19.3.0-4872.g0d61375b.el9. 789 kB/s | 175 kB 00:00 2024-09-11T14:40:33.649 INFO:teuthology.orchestra.run.smithi050.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 288 kB/s | 183 kB 00:00 2024-09-11T14:40:33.724 INFO:teuthology.orchestra.run.smithi050.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 700 kB/s | 513 kB 00:00 2024-09-11T14:40:33.841 INFO:teuthology.orchestra.run.smithi050.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-11T14:40:33.949 INFO:teuthology.orchestra.run.smithi050.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 590 kB/s | 64 kB 00:00 2024-09-11T14:40:33.991 INFO:teuthology.orchestra.run.smithi050.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2024-09-11T14:40:34.058 INFO:teuthology.orchestra.run.smithi050.stdout:(8/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-09-11T14:40:35.234 INFO:teuthology.orchestra.run.smithi050.stdout:(9/9): ceph-test-19.3.0-4872.g0d61375b.el9.x86_ 35 MB/s | 85 MB 00:02 2024-09-11T14:40:35.237 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:40:35.237 INFO:teuthology.orchestra.run.smithi050.stdout:Total 29 MB/s | 87 MB 00:02 2024-09-11T14:40:35.324 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:40:35.350 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:40:35.350 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:40:35.983 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:40:35.984 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:40:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:40:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-11T14:40:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-11T14:40:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-11T14:40:37.138 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-11T14:40:37.171 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:40:37.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:40:37.287 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-11T14:40:37.331 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-11T14:40:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:40:49.286 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 8/9 2024-09-11T14:40:49.373 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:40:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 9/9 2024-09-11T14:40:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 1/9 2024-09-11T14:40:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2/9 2024-09-11T14:40:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-11T14:40:55.619 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-11T14:40:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-11T14:40:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-11T14:40:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-11T14:40:55.620 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-11T14:40:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-11T14:40:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:55.974 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:40:55.974 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:55.975 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:40:56.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-09-11T14:40:56.912 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:40:57.055 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 6.5 k 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 2.2 M 2024-09-11T14:40:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 1.6 M 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 291 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 5.1 M 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 71 M 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T14:40:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T14:40:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:40:57.061 INFO:teuthology.orchestra.run.smithi050.stdout:Install 37 Packages 2024-09-11T14:40:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:40:57.062 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 84 M 2024-09-11T14:40:57.062 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 896 M 2024-09-11T14:40:57.062 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:40:57.532 INFO:teuthology.orchestra.run.smithi050.stdout:(1/37): ceph-19.3.0-4872.g0d61375b.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-11T14:40:57.732 INFO:teuthology.orchestra.run.smithi050.stdout:(2/37): ceph-mgr-19.3.0-4872.g0d61375b.el9.x86_ 4.9 MB/s | 1.6 MB 00:00 2024-09-11T14:40:57.782 INFO:teuthology.orchestra.run.smithi050.stdout:(3/37): ceph-mds-19.3.0-4872.g0d61375b.el9.x86_ 5.9 MB/s | 2.2 MB 00:00 2024-09-11T14:40:57.874 INFO:teuthology.orchestra.run.smithi050.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4872.g0d61 3.1 MB/s | 291 kB 00:00 2024-09-11T14:40:57.958 INFO:teuthology.orchestra.run.smithi050.stdout:(5/37): ceph-mon-19.3.0-4872.g0d61375b.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-09-11T14:40:58.075 INFO:teuthology.orchestra.run.smithi050.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-11T14:40:58.117 INFO:teuthology.orchestra.run.smithi050.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-11T14:40:58.179 INFO:teuthology.orchestra.run.smithi050.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 5.6 MB/s | 1.2 MB 00:00 2024-09-11T14:40:58.204 INFO:teuthology.orchestra.run.smithi050.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.5 MB/s | 135 kB 00:00 2024-09-11T14:40:58.229 INFO:teuthology.orchestra.run.smithi050.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-11T14:40:58.254 INFO:teuthology.orchestra.run.smithi050.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-11T14:40:58.388 INFO:teuthology.orchestra.run.smithi050.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-11T14:40:58.413 INFO:teuthology.orchestra.run.smithi050.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-11T14:40:58.439 INFO:teuthology.orchestra.run.smithi050.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 690 kB/s | 35 kB 00:00 2024-09-11T14:40:58.464 INFO:teuthology.orchestra.run.smithi050.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-11T14:40:58.489 INFO:teuthology.orchestra.run.smithi050.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-11T14:40:58.515 INFO:teuthology.orchestra.run.smithi050.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-11T14:40:58.557 INFO:teuthology.orchestra.run.smithi050.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-11T14:40:58.582 INFO:teuthology.orchestra.run.smithi050.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-11T14:40:58.607 INFO:teuthology.orchestra.run.smithi050.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-11T14:40:58.641 INFO:teuthology.orchestra.run.smithi050.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-11T14:40:58.666 INFO:teuthology.orchestra.run.smithi050.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 126 kB/s | 22 kB 00:00 2024-09-11T14:40:58.691 INFO:teuthology.orchestra.run.smithi050.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-09-11T14:40:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 909 kB/s | 23 kB 00:00 2024-09-11T14:40:58.750 INFO:teuthology.orchestra.run.smithi050.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 846 kB/s | 19 kB 00:00 2024-09-11T14:40:58.801 INFO:teuthology.orchestra.run.smithi050.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-09-11T14:40:58.826 INFO:teuthology.orchestra.run.smithi050.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-11T14:40:58.851 INFO:teuthology.orchestra.run.smithi050.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-11T14:40:58.877 INFO:teuthology.orchestra.run.smithi050.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-11T14:40:58.902 INFO:teuthology.orchestra.run.smithi050.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 654 kB/s | 16 kB 00:00 2024-09-11T14:40:58.927 INFO:teuthology.orchestra.run.smithi050.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-11T14:40:58.961 INFO:teuthology.orchestra.run.smithi050.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-11T14:40:58.986 INFO:teuthology.orchestra.run.smithi050.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-09-11T14:40:59.036 INFO:teuthology.orchestra.run.smithi050.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-11T14:40:59.053 INFO:teuthology.orchestra.run.smithi050.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T14:40:59.079 INFO:teuthology.orchestra.run.smithi050.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-11T14:41:00.304 INFO:teuthology.orchestra.run.smithi050.stdout:(37/37): ceph-osd-19.3.0-4872.g0d61375b.el9.x86 27 MB/s | 71 MB 00:02 2024-09-11T14:41:00.308 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:00.308 INFO:teuthology.orchestra.run.smithi050.stdout:Total 26 MB/s | 84 MB 00:03 2024-09-11T14:41:00.465 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:00.511 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:00.511 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:01.361 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:01.361 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:03.054 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:03.115 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T14:41:03.148 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T14:41:03.176 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T14:41:03.251 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T14:41:03.300 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T14:41:03.333 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T14:41:03.429 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T14:41:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T14:41:03.553 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T14:41:03.587 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T14:41:03.625 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T14:41:03.660 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T14:41:03.693 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T14:41:03.734 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T14:41:03.777 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T14:41:03.811 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T14:41:03.864 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T14:41:03.903 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T14:41:03.979 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T14:41:04.132 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T14:41:04.223 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T14:41:04.282 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T14:41:04.328 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T14:41:04.387 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T14:41:04.450 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T14:41:04.647 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T14:41:04.698 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T14:41:04.779 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T14:41:04.877 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T14:41:04.914 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T14:41:05.049 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T14:41:05.252 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 32/37 2024-09-11T14:41:05.282 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 33/37 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T14:41:05.310 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:16.499 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:41:16.528 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 34/37 2024-09-11T14:41:16.529 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:41:16.529 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T14:41:16.529 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:41:16.529 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T14:41:16.529 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:17.100 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:41:17.125 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 35/37 2024-09-11T14:41:17.126 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:41:17.126 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T14:41:17.126 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:41:17.126 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T14:41:17.126 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:17.376 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 36/37 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T14:41:17.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:17.468 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 37/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 1/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 3/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 4/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 5/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9. 6/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T14:41:24.119 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T14:41:24.120 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T14:41:24.121 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T14:41:24.122 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T14:41:24.123 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T14:41:24.123 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T14:41:24.123 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:25.225 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T14:41:25.226 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T14:41:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:25.228 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:25.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-09-11T14:41:26.196 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:26.290 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:41:26.334 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:26.335 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:41:26.335 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:26.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-09-11T14:41:26.989 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:27.128 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 784 k 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:27.129 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 784 k 2024-09-11T14:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 791 k 2024-09-11T14:41:27.130 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:27.393 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-19.3.0-4872.g0d61375b.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-11T14:41:27.394 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:27.394 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-11T14:41:27.394 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:27.400 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:27.400 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:27.406 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:27.406 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:27.478 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:27.909 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:41:27.992 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:41:28.391 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:41:28.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:41:28.631 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:28.631 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:28.632 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:28.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:28.632 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:28.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T14:41:29.468 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:29.605 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:29.605 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:29.605 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-11T14:41:29.605 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 158 k 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 158 k 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 644 k 2024-09-11T14:41:29.606 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:29.817 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-19.3.0-4872.g0d6137 750 kB/s | 158 kB 00:00 2024-09-11T14:41:29.818 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:29.818 INFO:teuthology.orchestra.run.smithi050.stdout:Total 746 kB/s | 158 kB 00:00 2024-09-11T14:41:29.818 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:29.829 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:29.829 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:29.891 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:29.891 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:41:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:41:30.137 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:41:30.137 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T14:41:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b. 1/1 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:31.123 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:31.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-09-11T14:41:31.988 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:29 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:32.081 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:41:32.125 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:32.127 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:41:32.127 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:32.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T14:41:32.798 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:32.939 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 14 M 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T14:41:32.940 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 34 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 16 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout:Install 11 Packages 2024-09-11T14:41:32.941 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:32.942 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 18 M 2024-09-11T14:41:32.942 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 155 M 2024-09-11T14:41:32.942 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:33.562 INFO:teuthology.orchestra.run.smithi050.stdout:(1/11): ceph-grafana-dashboards-19.3.0-4872.g0d 226 kB/s | 34 kB 00:00 2024-09-11T14:41:33.587 INFO:teuthology.orchestra.run.smithi050.stdout:(2/11): ceph-prometheus-alerts-19.3.0-4872.g0d6 93 kB/s | 16 kB 00:00 2024-09-11T14:41:33.854 INFO:teuthology.orchestra.run.smithi050.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 923 kB/s | 269 kB 00:00 2024-09-11T14:41:33.905 INFO:teuthology.orchestra.run.smithi050.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T14:41:33.930 INFO:teuthology.orchestra.run.smithi050.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-11T14:41:34.106 INFO:teuthology.orchestra.run.smithi050.stdout:(6/11): ceph-mgr-dashboard-19.3.0-4872.g0d61375 20 MB/s | 14 MB 00:00 2024-09-11T14:41:34.164 INFO:teuthology.orchestra.run.smithi050.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.8 MB/s | 2.0 MB 00:00 2024-09-11T14:41:34.189 INFO:teuthology.orchestra.run.smithi050.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-11T14:41:34.214 INFO:teuthology.orchestra.run.smithi050.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-09-11T14:41:34.240 INFO:teuthology.orchestra.run.smithi050.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-11T14:41:34.698 INFO:teuthology.orchestra.run.smithi050.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 777 kB/s | 863 kB 00:01 2024-09-11T14:41:34.704 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:34.704 INFO:teuthology.orchestra.run.smithi050.stdout:Total 10 MB/s | 18 MB 00:01 2024-09-11T14:41:34.737 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:34.759 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:34.760 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:35.054 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:35.055 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:35.622 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:35.715 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-11T14:41:35.743 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-11T14:41:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-11T14:41:36.000 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-11T14:41:36.072 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-11T14:41:36.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-11T14:41:36.208 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-11T14:41:36.236 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:41:36.277 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 9/11 2024-09-11T14:41:39.651 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 10/11 2024-09-11T14:41:39.706 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:41:41.566 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 11/11 2024-09-11T14:41:41.566 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el 1/11 2024-09-11T14:41:41.566 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noa 2/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9 3/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-11T14:41:41.567 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-11T14:41:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-11T14:41:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:41.880 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:41.880 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:41.881 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:42.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T14:41:42.751 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:42.891 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:42.892 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-11T14:41:42.892 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 7.4 M 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T14:41:42.893 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-11T14:41:42.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:42.895 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-09-11T14:41:42.895 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 212 M 2024-09-11T14:41:42.895 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:43.205 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-11T14:41:43.321 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-11T14:41:43.416 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-11T14:41:43.466 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 153 kB/s | 18 kB 00:00 2024-09-11T14:41:43.556 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-48 14 MB/s | 7.4 MB 00:00 2024-09-11T14:41:43.581 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 337 kB/s | 38 kB 00:00 2024-09-11T14:41:43.681 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-11T14:41:43.815 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 19 MB/s | 4.9 MB 00:00 2024-09-11T14:41:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-11T14:41:43.948 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-11T14:41:44.016 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 4.3 MB/s | 3.0 MB 00:00 2024-09-11T14:41:44.299 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-11T14:41:44.300 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:44.300 INFO:teuthology.orchestra.run.smithi050.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-11T14:41:44.667 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:44.694 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:44.694 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:45.153 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:45.154 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T14:41:46.048 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:41:46.152 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:41:46.191 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T14:41:46.385 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T14:41:46.431 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T14:41:46.511 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T14:41:47.197 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T14:41:47.314 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T14:41:48.646 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:41:49.733 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T14:41:49.763 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:41:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 12/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d613 1/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T14:41:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T14:41:52.781 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T14:41:52.781 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:52.781 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T14:41:52.782 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T14:41:52.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:52.783 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:53.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-09-11T14:41:53.788 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:53.928 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 49 k 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T14:41:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:53.931 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:53.932 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-09-11T14:41:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:53.932 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-09-11T14:41:53.932 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-09-11T14:41:53.932 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:54.656 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 213 kB/s | 19 kB 00:00 2024-09-11T14:41:54.681 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 26 kB 00:00 2024-09-11T14:41:54.740 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): ceph-mgr-rook-19.3.0-4872.g0d61375b.el9 284 kB/s | 49 kB 00:00 2024-09-11T14:41:54.765 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-11T14:41:54.816 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-11T14:41:54.866 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 535 kB/s | 54 kB 00:00 2024-09-11T14:41:54.908 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 351 kB/s | 32 kB 00:00 2024-09-11T14:41:54.933 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-11T14:41:54.958 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-11T14:41:54.992 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-11T14:41:55.017 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-11T14:41:55.042 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-11T14:41:55.084 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-11T14:41:55.087 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:41:55.088 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-11T14:41:55.157 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:41:55.174 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:41:55.174 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:41:55.273 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:41:55.280 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:41:55.801 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:41:55.977 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T14:41:56.030 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T14:41:56.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T14:41:56.121 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T14:41:56.163 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T14:41:56.271 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T14:41:56.390 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T14:41:56.440 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T14:41:57.231 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T14:41:57.278 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T14:41:57.312 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T14:41:57.361 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T14:41:57.389 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 13/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 1/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T14:41:58.160 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T14:41:58.161 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T14:41:58.161 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T14:41:58.161 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T14:41:58.161 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T14:41:58.161 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T14:41:58.502 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T14:41:58.502 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:41:58.502 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T14:41:58.503 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T14:41:58.504 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T14:41:58.504 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T14:41:58.504 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T14:41:58.504 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T14:41:58.504 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T14:41:58.505 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T14:41:58.505 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:58.505 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:41:58.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T14:41:59.396 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:41:59.537 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 154 k 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T14:41:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-09-11T14:41:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:41:59.540 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-09-11T14:41:59.540 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-09-11T14:41:59.540 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:41:59.903 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4872.g0d61375b.e 728 kB/s | 154 kB 00:00 2024-09-11T14:41:59.961 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-11T14:41:59.987 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T14:42:00.012 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T14:42:00.270 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 428 kB/s | 248 kB 00:00 2024-09-11T14:42:00.671 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.1 MB/s | 6.0 MB 00:00 2024-09-11T14:42:00.673 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:00.673 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.2 MB/s | 7.0 MB 00:01 2024-09-11T14:42:00.748 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:00.760 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:00.760 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:00.869 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:00.869 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:01.184 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:01.370 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T14:42:01.432 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T14:42:01.987 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T14:42:02.078 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T14:42:02.154 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T14:42:02.185 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:42:02.964 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 6/6 2024-09-11T14:42:02.964 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 1/6 2024-09-11T14:42:02.964 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T14:42:02.964 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T14:42:02.964 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T14:42:02.965 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T14:42:03.245 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T14:42:03.245 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:03.246 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:03.246 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:42:03.246 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T14:42:03.246 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T14:42:03.246 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T14:42:03.247 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T14:42:03.247 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T14:42:03.247 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:03.247 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:03.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-09-11T14:42:04.093 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:04.233 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:04.233 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:04.233 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 884 k 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-09-11T14:42:04.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:04.235 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 964 k 2024-09-11T14:42:04.235 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 4.1 M 2024-09-11T14:42:04.235 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:04.639 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-fuse-19.3.0-4872.g0d61375b.el9.x86_ 3.1 MB/s | 884 kB 00:00 2024-09-11T14:42:04.823 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 171 kB/s | 80 kB 00:00 2024-09-11T14:42:04.823 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:04.824 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.6 MB/s | 964 kB 00:00 2024-09-11T14:42:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:04.861 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:04.861 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:04.946 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:04.947 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:05.289 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T14:42:05.359 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:42:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2/2 2024-09-11T14:42:06.024 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/2 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:06.303 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:06.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-09-11T14:42:07.230 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:05 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:07.371 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.3.0-4872.g0d61375b.el9 ceph-noarch 285 k 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:07.372 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 285 k 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.3 M 2024-09-11T14:42:07.373 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:07.587 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-19.3.0-4872.g0d61375b.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-11T14:42:07.588 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:07.588 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-11T14:42:07.588 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:07.597 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:07.597 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:07.634 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:07.634 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:07.866 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:07.938 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:42:07.955 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:42:07.955 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:42:07.955 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T14:42:07.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:08.944 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 1/1 2024-09-11T14:42:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:08.945 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.3.0-4872.g0d61375b.el9.noarch 2024-09-11T14:42:08.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:08.945 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:09.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-09-11T14:42:09.874 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:07 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:10.014 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:10.015 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:10.015 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-11T14:42:10.015 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:10.015 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 125 k 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 125 k 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 445 k 2024-09-11T14:42:10.016 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:10.208 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-19.3.0-4872.g0d61375b.el9.x86_64 650 kB/s | 125 kB 00:00 2024-09-11T14:42:10.209 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:10.209 INFO:teuthology.orchestra.run.smithi050.stdout:Total 645 kB/s | 125 kB 00:00 2024-09-11T14:42:10.209 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:10.215 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:10.215 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:10.283 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:10.283 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:10.440 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:10.559 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:11.029 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:11.302 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:11.302 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:11.303 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:11.303 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:42:11.303 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:11.303 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:11.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-09-11T14:42:12.207 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:10 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:12.301 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:12.347 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:12.348 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:12.348 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:12.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-09-11T14:42:13.025 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:13.165 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:13.166 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 32 k 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 32 k 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 142 k 2024-09-11T14:42:13.167 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:13.329 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-19.3.0-4872.g0d61375b.el9.x86_6 196 kB/s | 32 kB 00:00 2024-09-11T14:42:13.330 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:13.330 INFO:teuthology.orchestra.run.smithi050.stdout:Total 195 kB/s | 32 kB 00:00 2024-09-11T14:42:13.330 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:13.333 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:13.334 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:13.351 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:13.351 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:13.467 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:13.596 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:14.003 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:14.242 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:14.242 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:42:14.243 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:14.243 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:14.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-09-11T14:42:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:15.127 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:15.175 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:15.176 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:15.176 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:15.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-09-11T14:42:15.844 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:15.939 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:15.984 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:15.985 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:15.985 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:16.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-09-11T14:42:16.649 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:14 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:16.742 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:16.789 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:16.790 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:16.790 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:16.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-09-11T14:42:17.457 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:15 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:17.551 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:17.598 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:17.598 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:17.599 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:17.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-09-11T14:42:18.266 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:16 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:18.360 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:18.405 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:18.406 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:18.406 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:18.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-09-11T14:42:19.067 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:19.160 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:19.3.0-4872.g0d61375b.el9.x86_64 is already installed. 2024-09-11T14:42:19.207 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:19.208 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-11T14:42:19.208 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:19.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-09-11T14:42:19.872 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:20.014 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:20.014 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:20.014 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-11T14:42:20.014 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:20.014 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 91 k 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:20.015 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 91 k 2024-09-11T14:42:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 321 k 2024-09-11T14:42:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:20.332 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-19.3.0-4872.g0d61375b.el9.x86_64.rpm 286 kB/s | 91 kB 00:00 2024-09-11T14:42:20.332 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:20.333 INFO:teuthology.orchestra.run.smithi050.stdout:Total 285 kB/s | 91 kB 00:00 2024-09-11T14:42:20.333 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:20.343 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:20.343 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:20.411 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:20.412 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:20.567 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:20.674 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:21.096 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:21.377 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:21.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-09-11T14:42:22.283 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:20 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 3.5 M 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.5 M 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 30 M 2024-09-11T14:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:23.895 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-19.3.0-4872.g0d61375b.el9.x86_64.rpm 2.4 MB/s | 3.5 MB 00:01 2024-09-11T14:42:23.896 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:23.896 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.4 MB/s | 3.5 MB 00:01 2024-09-11T14:42:23.896 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:23.911 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:23.911 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:24.002 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:24.002 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:24.619 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:24.713 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:24.740 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:24.740 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T14:42:24.740 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T14:42:24.740 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:42:24.740 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T14:42:24.741 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:25.974 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:25.974 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:25.974 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:25.974 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:42:25.974 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:25.975 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:26.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-09-11T14:42:26.846 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:24 ago on Wed 11 Sep 2024 02:40:02 PM UTC. 2024-09-11T14:42:26.986 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-11T14:42:26.986 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:26.986 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-11T14:42:26.986 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.3.0-4872.g0d61375b.el9 ceph 177 k 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 177 k 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 685 k 2024-09-11T14:42:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-11T14:42:27.199 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-19.3.0-4872.g0d61375b.el9.x86_64.rpm 836 kB/s | 177 kB 00:00 2024-09-11T14:42:27.200 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-11T14:42:27.200 INFO:teuthology.orchestra.run.smithi050.stdout:Total 829 kB/s | 177 kB 00:00 2024-09-11T14:42:27.200 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-11T14:42:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-11T14:42:27.211 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-11T14:42:27.279 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-11T14:42:27.280 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-11T14:42:27.447 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-11T14:42:27.559 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:28.008 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 1/1 2024-09-11T14:42:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:28.284 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-11T14:42:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.3.0-4872.g0d61375b.el9.x86_64 2024-09-11T14:42:28.284 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:28.285 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-11T14:42:28.615 DEBUG:teuthology.parallel:result is None 2024-09-11T14:42:28.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-11T14:42:28.757 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T14:42:28.780 INFO:teuthology.orchestra.run.smithi050.stdout:19.3.0-4872.g0d61375b.el9 2024-09-11T14:42:28.781 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4872.g0d61375b.el9 2024-09-11T14:42:28.781 INFO:teuthology.task.install:The correct ceph version 19.3.0-4872.g0d61375b is installed. 2024-09-11T14:42:28.783 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T14:42:28.783 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:28.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T14:42:28.847 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T14:42:28.847 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:28.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T14:42:28.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T14:42:28.980 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T14:42:28.980 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:28.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T14:42:29.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T14:42:29.111 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T14:42:29.111 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:29.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T14:42:29.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T14:42:29.243 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T14:42:29.329 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T14:42:29.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-09-11T14:42:29.355 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T14:42:29.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T14:42:29.421 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T14:42:29.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T14:42:29.495 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T14:42:29.495 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T14:42:29.496 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373', 'branch': 'main', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_cls_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7901373', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'workunit': {'branch': 'wip-matanb-crimson-seastore-logs', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 5813, 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-seastore-logs', 'suite_path': '/home/teuthworker/src/github.com_matan-b_ceph_dbebf1a3db6db7b5579464ee715edcf1b3e9cce2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/matan-b/ceph.git', 'suite_sha1': 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvIoVkVwDQBXSOxXJnqBzu3AVe9GMpZeH4YI7axkJA9vponZ7MFXR1XCxDjBDxn7sRmFzV6sdR3nPOC+ruIx2o='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-11_13:56:22', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T14:42:29.496 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T14:42:29.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T14:42:29.611 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:29.611 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T14:42:29.667 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T14:42:29.668 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:42:29.723 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-11 14:42:24.988009485 +0000 2024-09-11T14:42:29.724 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-11 14:39:24.888232840 +0000 2024-09-11T14:42:29.724 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-11 14:39:24.888232840 +0000 2024-09-11T14:42:29.724 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-11 14:39:24.888232840 +0000 2024-09-11T14:42:29.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T14:42:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-11T14:42:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-11T14:42:29.787 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.7157e-05 s, 5.9 MB/s 2024-09-11T14:42:29.788 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T14:42:29.845 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-11 14:42:24.989009459 +0000 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-11 14:39:25.222224137 +0000 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-11 14:39:25.222224137 +0000 2024-09-11T14:42:29.901 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-11 14:39:25.222224137 +0000 2024-09-11T14:42:29.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T14:42:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-11T14:42:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-11T14:42:29.965 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000115399 s, 4.4 MB/s 2024-09-11T14:42:29.966 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T14:42:30.024 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-09-11T14:42:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-11 14:42:24.989009459 +0000 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-11 14:39:25.529216137 +0000 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-11 14:39:25.529216137 +0000 2024-09-11T14:42:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-11 14:39:25.529216137 +0000 2024-09-11T14:42:30.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T14:42:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-11T14:42:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-11T14:42:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00010719 s, 4.8 MB/s 2024-09-11T14:42:30.145 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T14:42:30.202 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-11 14:42:24.989009459 +0000 2024-09-11T14:42:30.259 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-11 14:39:25.867207329 +0000 2024-09-11T14:42:30.260 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-11 14:39:25.867207329 +0000 2024-09-11T14:42:30.260 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-11 14:39:25.867207329 +0000 2024-09-11T14:42:30.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T14:42:30.323 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-11T14:42:30.323 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-11T14:42:30.323 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.5625e-05 s, 6.0 MB/s 2024-09-11T14:42:30.324 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T14:42:30.381 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:42:30.381 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:42:30.381 INFO:tasks.ceph:Generating config... 2024-09-11T14:42:30.382 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T14:42:30.383 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-11T14:42:30.383 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-11T14:42:30.383 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-11T14:42:30.383 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-11T14:42:30.383 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T14:42:30.383 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-11T14:42:30.384 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-11T14:42:30.385 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-11T14:42:30.385 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-11T14:42:30.385 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-11T14:42:30.385 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T14:42:30.385 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T14:42:30.385 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-09-11T14:42:30.386 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-09-11T14:42:30.386 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-09-11T14:42:30.386 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T14:42:30.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T14:42:30.481 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T14:42:30.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T14:42:30.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:42:30.562 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50')] 2024-09-11T14:42:30.563 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-09-11T14:42:30.564 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:30.564 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:42:30.618 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:30.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:30.710+0000 7f58296d7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:30.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:30.710+0000 7f58296d7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:30.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:30.710+0000 7f58296d7f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 85b31bab-bc01-45a8-9735-fc6a813abd91 2024-09-11T14:42:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-09-11T14:42:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2024-09-11T14:42:30.713 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 85b31bab-bc01-45a8-9735-fc6a813abd91 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2024-09-11T14:42:30.712477+0000 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-09-11T14:42:30.712477+0000 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 17 (quincy) 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-09-11T14:42:30.714 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-11T14:42:30.715 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T14:42:30.730 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 85b31bab-bc01-45a8-9735-fc6a813abd91... 2024-09-11T14:42:30.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T14:42:30.815 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-09-11T14:42:30.815 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-09-11T14:42:30.816 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-09-11T14:42:30.817 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.818 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.50 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms client mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms cluster mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon client mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon cluster mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon service mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: ms service mode = crc 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 85b31bab-bc01-45a8-9735-fc6a813abd91 2024-09-11T14:42:30.819 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-09-11T14:42:30.820 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug trim objects = true 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-09-11T14:42:30.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: crimson alien thread cpu cores = 6-7 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: crimson osd obc lru size = 10 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore backref = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore cache = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore cleaner = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore device = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore epm = 20 2024-09-11T14:42:30.822 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore fixedkv tree = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore journal = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore lba = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore odata = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore ompap = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore onode = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore t = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug seastore tm = 20 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = seastore 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-11T14:42:30.823 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-09-11T14:42:30.824 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.825 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon min osdmap epochs = 50 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune interval = 2 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune min = 15 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune txsize = 2 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: paxos service trim min = 10 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default crimson = True 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-09-11T14:42:30.826 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: crimson seastar cpu cores = 0-1 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: crimson seastar cpu cores = 2-3 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-09-11T14:42:30.827 INFO:teuthology.orchestra.run.smithi050.stdout: crimson seastar cpu cores = 4-5 2024-09-11T14:42:30.828 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T14:42:30.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T14:42:30.895 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T14:42:30.895 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:30.895 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T14:42:30.911 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:30.911 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T14:42:30.966 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:42:30.967 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:30.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T14:42:30.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T14:42:31.043 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:31.043 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:31.099 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T14:42:31.099 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T14:42:31.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:42:31.200 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T14:42:31.203 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T14:42:31.203 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T14:42:31.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T14:42:31.256 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T14:42:31.269 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T14:42:31.269 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T14:42:31.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T14:42:31.295 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:42:31.295 INFO:tasks.ceph:role: osd.0 2024-09-11T14:42:31.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:42:31.295 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T14:42:31.360 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:42:31.360 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:42:31.360 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:42:31.360 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:42:31.361 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:42:31.392 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-11T14:42:31.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-11T14:42:31.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T14:42:31.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T14:42:31.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T14:42:31.602 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:42:31.602 INFO:tasks.ceph:role: osd.1 2024-09-11T14:42:31.602 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:42:31.602 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:42:31.669 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:42:31.700 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-11T14:42:31.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-11T14:42:31.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T14:42:31.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T14:42:31.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T14:42:31.908 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T14:42:31.908 INFO:tasks.ceph:role: osd.2 2024-09-11T14:42:31.908 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:42:31.909 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T14:42:31.973 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T14:42:31.974 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T14:42:32.006 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-11T14:42:32.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-11T14:42:32.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T14:42:32.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T14:42:32.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:32.459 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:32.460 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:32,460 seastar - Reactor backend: linux-aio 2024-09-11T14:42:32.472 INFO:teuthology.orchestra.run.smithi050.stderr:==27823==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:32.605 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:32,605 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:42:32.606 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:32,605 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-1 2024-09-11T14:42:32.606 INFO:teuthology.orchestra.run.smithi050.stderr:DEBUG 2024-09-11 14:42:32,606 [shard 0:main] seastar - reactor::drain 2024-09-11T14:42:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:33.158 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:33.158 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:33.159 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:33.213 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:33,217 seastar - Reactor backend: linux-aio 2024-09-11T14:42:33.225 INFO:teuthology.orchestra.run.smithi050.stderr:==27821==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:33.290 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4aba4 0x5ff0dd6 0x5ff0fd2 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:33.290 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x76e232a 0xba2793e 0xba323fd 0xba325c9 0xb9d96db 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.585 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3630302 0x3630a72 0xac9c6e3 0xad02411 0xaccefd1 0x994da0a 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:33.586 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.611 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x159824 0xc928c 0xc9e39 0x360e4e8 0x3b92070 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f4222 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:33.611 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x42198 0x3ef7b 0xfd531 0xba2d45f 0xb9d2631 0x9b06dc0 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:33.669 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff921bed50 0xffffffff921bf6b7 0xffffffff921e4853 0xffffffff9218e356 0xffffffff92196a3b 0xffffffff92196ebc 0xffffffff91e8c884 0xffffffff92ad1d62 0xffffffff92c00bb2 2024-09-11T14:42:33.706 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xfd51f 0xb85ab4d 0xb85ac00 0xb85ad16 0xb85adf6 0xb85b076 0xb85b12f 0xb85b621 0xb85ba2c 0xb85bceb 0xba37fd0 0xba3826c 0xba38433 0xba386a7 0xba387b0 0xba3892c 0xba390d8 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:33.707 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.764 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab49 0x5ff0dd6 0x5ff0f9e 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5e1a2dd 0x5e1b192 0x4f18f39 0x4f191ec 0x959a84e 0xac9c698 0xad02411 0xaccefd1 0x994da0a 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.816 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c54 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:33.816 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.873 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10798a 0xdd2e1 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:33.874 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:33.909 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549131 0x554a11c 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:33.910 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:35.557 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:35.557 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:35.557 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:35.558 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:35.558 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:35.558 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:35.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T14:42:35.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:35.907 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:35.908 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:35,908 seastar - Reactor backend: linux-aio 2024-09-11T14:42:35.913 INFO:teuthology.orchestra.run.smithi050.stderr:==27868==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:36.021 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4ab7c 0x5ff0dd6 0x5ff0fd2 0x5ff11cf 0x5ff14a5 0x603fba3 0x603fda9 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x3968bb0 0x396a38a 0x380ac18 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:36.021 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:36.040 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:36,040 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:42:36.041 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:36,040 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 2-3 2024-09-11T14:42:36.041 INFO:teuthology.orchestra.run.smithi050.stderr:DEBUG 2024-09-11 14:42:36,041 [shard 0:main] seastar - reactor::drain 2024-09-11T14:42:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:36.573 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:36.573 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:36.574 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:36.633 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:36.637 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:36,636 seastar - Reactor backend: linux-aio 2024-09-11T14:42:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:==27866==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4b14c 0x5fde990 0x5fe4933 0x5fe4af5 0x603fd14 0x5fbfef6 0x5fc042a 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4b14c 0x3b920e7 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f4222 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.013 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133c54 0xdd332 0x4ab1c 0x3acdafe 0x3b9225d 0xb9f768b 0xb9fb0f9 0xba3eb1c 0xb9e10b0 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9e09e02 0x9e2ca55 0x999b8a7 0x994bbe6 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:37.013 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x10d778 0xdd31a 0x4ab1c 0x611b8e4 0xba3becf 0xba3bfdb 0xba3c1ec 0xba3c388 0xba3c71e 0xba3c930 0xb9d97af 0xb9d9bd4 0xb9dfb97 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:37.039 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.096 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b587 0x10d76c 0xdd31a 0x4ab1c 0x3acdafe 0x3b9225d 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f45e7 0xb9e014c 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b05f61 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:37.097 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.134 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x1597a0 0xc928c 0xc9e39 0x360e4e8 0x3b92070 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:37.135 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.192 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5c9433b 0xb9f1bbb 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b05f61 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:37.192 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.244 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x9e3b019 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:37.244 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.301 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abd0 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:37.301 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:37.340 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x4abd0 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:37.340 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:38.989 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:38.989 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:38.989 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:38.989 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:39.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T14:42:39.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:39.334 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:39,335 seastar - Reactor backend: linux-aio 2024-09-11T14:42:39.341 INFO:teuthology.orchestra.run.smithi050.stderr:==27913==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:39,473 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-09-11T14:42:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:39,473 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 4-5 2024-09-11T14:42:39.474 INFO:teuthology.orchestra.run.smithi050.stderr:DEBUG 2024-09-11 14:42:39,474 [shard 0:main] seastar - reactor::drain 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:40.007 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:40.065 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-11T14:42:40.069 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2024-09-11 14:42:40,069 seastar - Reactor backend: linux-aio 2024-09-11T14:42:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:==27911==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-11T14:42:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xdd2d8 0x4b14c 0x5fde990 0x5fe4933 0x5fe4af5 0x603fd14 0x5fbfef6 0x5fc00d0 0x5fc065c 0x5e081e5 0x5e20427 0x715ebc2 0x7170395 0x7171832 0x71718b7 0x71719b9 0x7171ab1 0x7171cbe 0x7171ea4 0x71721f3 0x7172597 0x71727c7 0x7172a31 0x7172cfe 0x7172f1c 0x7172fd4 0x717309a 0x717317b 0x5c6b026 0x5c6b130 0x5c6b1ec 0x5c6b2a8 0x5c6b364 0x5c6b4b1 0x5c6b63c 0x5c6b9fd 0x5c85800 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:40.148 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xc9277 0xc9e39 0x360e4e8 0x3b92070 0x3b92268 0xb9f075a 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xb9f3dd7 0xb9f3fd3 0xba38ebd 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.443 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0xba3bff1 0xba3c1ec 0xba3c388 0xba3c71e 0xba3c930 0xb9d97af 0xb9d9bd4 0xb9e163f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9e09e02 0x9e2ca55 0x999b8a7 0x994bbe6 0x927bd5d 0x93356ad 0x933664c 0x93366cf 0x9336754 0x9336856 0x933694e 0x93369d4 0x9336a8d 0x9336c43 0x9336d7b 0x9336df6 0x9336e7b 0x9336fd6 0x93370f0 0x9337175 0x9337253 0x9559f46 0x9622cb3 0x9622e30 0x96e3b91 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:40.443 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.472 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x3b921b6 0x3b92268 0xb9f0927 0xb9f1bfa 0xb9f1e74 0xb9f201c 0xba39371 0xb9e038f 0xb9e3db8 0xb9ced56 0xb9cf13a 0x9b016e3 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:40.472 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.527 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x422da 0x3ef7b 0xfd531 0xb85ab4d 0xb85ac00 0xb85ad16 0xb85adf6 0xb85b4b6 0xb85b621 0xb85ba2c 0xb85bceb 0xb9e2239 0xb9ced56 0xb9cf13a 0x9b05f61 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:40.527 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.565 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x376ee1a 0xba179f0 0xb9d61ef 0x984ae51 0x9b0161a 0x9b19232 0x99a9828 0x994a011 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:40.565 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.623 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x15a3ff 0x422c4 0x3ef7b 0xfd531 0x9763504 0xb0ea919 0xb0eafc9 0xb0d4fc2 0xac9c741 0xad02411 0xaccefd1 0x994da0a 0x927bd5d 0x927c472 0x927c98e 0x927cb17 0x927cd5e 0x927d036 0x927d0eb 0x959a3a8 0x959a60d 0x95d25f3 0x96d6bde 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:40.623 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff92ae3f2b 0xffffffff92150214 0xffffffff92150308 0xffffffff9218ed15 0xffffffff92196a3b 0xffffffff92196ebc 0xffffffff91e8c884 0xffffffff92ad1d62 0xffffffff92c00bb2 2024-09-11T14:42:40.674 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x5549ab8 0x9e3b03b 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:40.674 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.732 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x133cd1 0xdd332 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc636bd 0xb931f8d 0xb9327c5 0x380b428 0x2958f 0x2963f 0x35f29a4 2024-09-11T14:42:40.732 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbcafa0f 0xbb88399 0xbb89904 0xbb89b1a 0xbb89c70 0xbb8a139 0x3e6ef 0x8b568 0x10d76c 0xdd31a 0x4ab1c 0x554a17b 0x9e3b000 0x9eaa4f9 0x9edacf1 0x9e42284 0x9960817 0x987fc22 0x98a42e2 0x98a4441 0x98b5cde 0x98b5eae 0x98b61bc 0x98b64cb 0x98b662a 0x98b66a5 0x98b68ec 0x98b6bb0 0x98b6c40 0x98b6ce4 0x95cab72 0x95cadd7 0x95d337c 0x96c5a26 0xbb9c947 0xbbb47c5 0xbc6206f 0xbc7466b 0xbc74ced 0x7b37e35 0xc60a404 0x5e669 0x89c51 0x10ec7f 2024-09-11T14:42:40.768 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2024-09-11T14:42:42.434 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:42.434 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2024-09-11T14:42:42.434 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2024-09-11T14:42:42.435 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2024-09-11T14:42:42.435 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2024-09-11T14:42:42.435 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:42:42.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T14:42:42.486 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T14:42:42.487 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:42.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T14:42:42.552 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:42.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T14:42:42.618 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:42.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T14:42:42.684 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:42.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T14:42:42.749 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:42.749 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T14:42:42.805 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T14:42:42.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBXrOFmNVTvCxAAWpikLbU06VJvBmjWhuYQ2g== 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBZrOFmETjaFhAAD+VCaioPIFgyZ9JWcEZsRg== 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBcrOFm9kBRMBAA2ZXshJLHig+JfDmVFTkAaQ== 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-09-11T14:42:42.869 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBgrOFmRq9gDhAAywoaOg/6dUP45alE6v2z+A== 2024-09-11T14:42:42.870 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-09-11T14:42:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQBXrOFm4o83DxAAbYRBxGYaZPFO9/CnCD8Qrw== 2024-09-11T14:42:42.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T14:42:42.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:42:43.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:42:43.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T14:42:43.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T14:42:43.230 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T14:42:43.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T14:42:43.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T14:42:43.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.429+0000 7fbe49266c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.429+0000 7fbe49266c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T14:42:43.470 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T14:42:43.526 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T14:42:43.526 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T14:42:43.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T14:42:43.570 INFO:tasks.ceph.mon.a:Started 2024-09-11T14:42:43.570 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T14:42:43.570 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T14:42:43.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T14:42:43.573 INFO:tasks.ceph.mgr.x:Started 2024-09-11T14:42:43.573 DEBUG:tasks.ceph:set 0 configs 2024-09-11T14:42:43.573 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T14:42:43.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T14:42:43.657 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-11T14:42:43.655+0000 7f37864dbc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.657 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-11T14:42:43.655+0000 7f37864dbc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.657 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-11T14:42:43.655+0000 7f37864dbc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.658 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-11T14:42:43.657+0000 7f5914e31000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.681+0000 7f60c1f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.693+0000 7f60c1f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.767 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-09-11T14:42:43.779 INFO:tasks.ceph:check_enable_crimson: True 2024-09-11T14:42:43.779 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-11T14:42:43.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-11T14:42:43.865 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-11T14:42:43.864+0000 7f5914e31000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.866 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-11T14:42:43.864+0000 7f5914e31000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.887+0000 7f3bf9711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:43.898 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:43.897+0000 7f3bf9711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:44.000 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-11T14:42:43.999+0000 7f377d1f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-11T14:42:44.001 INFO:tasks.ceph.mon.a.smithi050.stderr:Please be aware that this feature is experimental, untested, 2024-09-11T14:42:44.001 INFO:tasks.ceph.mon.a.smithi050.stderr:unsupported, and may result in data corruption, data loss, 2024-09-11T14:42:44.001 INFO:tasks.ceph.mon.a.smithi050.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-11T14:42:44.001 INFO:tasks.ceph.mon.a.smithi050.stderr:feature with important data. 2024-09-11T14:42:44.001 INFO:tasks.ceph.mon.a.smithi050.stderr: 2024-09-11T14:42:44.780 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T14:42:44.780 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:44.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T14:42:44.805 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:44.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T14:42:44.873 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:42:44.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T14:42:44.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 5469008b-2fe0-4f1f-9eb0-6ea59e2bab88 0 2024-09-11T14:42:45.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.084+0000 7f5ab247d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.095+0000 7f5ab247d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.178 INFO:teuthology.orchestra.run.smithi050.stdout:0 2024-09-11T14:42:45.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 88f3e561-4b0e-47fd-b4ac-552fe6bcbea3 1 2024-09-11T14:42:45.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.293+0000 7fce50e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.304+0000 7fce50e4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.386 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-09-11T14:42:45.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 31867ae1-289b-464a-a7b8-455848503331 2 2024-09-11T14:42:45.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.502+0000 7f2f0c951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.513+0000 7f2f0c951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.595 INFO:teuthology.orchestra.run.smithi050.stdout:2 2024-09-11T14:42:45.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-11T14:42:45.638 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T14:42:45.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-11T14:42:45.681 INFO:tasks.ceph.osd.0:Started 2024-09-11T14:42:45.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-11T14:42:45.713 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T14:42:45.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-11T14:42:45.756 INFO:tasks.ceph.osd.1:Started 2024-09-11T14:42:45.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-11T14:42:45.789 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T14:42:45.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-11T14:42:45.833 INFO:tasks.ceph.osd.2:Started 2024-09-11T14:42:45.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:42:45.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.958+0000 7f6d8d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:45.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:45.968+0000 7f6d8d26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:46.061 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:46.061 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":6,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:42:45.593752+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:42:46.074 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T14:42:46.075 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T14:42:46.376 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:42:46.485 INFO:teuthology.misc.health.smithi050.stderr:2024-09-11T14:42:46.480+0000 7f740d991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:46.489 INFO:teuthology.misc.health.smithi050.stderr:2024-09-11T14:42:46.488+0000 7f740d991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:46.561 INFO:teuthology.misc.health.smithi050.stdout: 2024-09-11T14:42:46.561 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":6,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:42:45.593752+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:42:46.572 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T14:42:49.391 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-11T14:42:49.390+0000 7f58b1c49640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T14:42:52.875 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T14:42:52.985 INFO:teuthology.misc.health.smithi050.stderr:2024-09-11T14:42:52.979+0000 7f0423d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:52.989 INFO:teuthology.misc.health.smithi050.stderr:2024-09-11T14:42:52.988+0000 7f0423d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.192 INFO:teuthology.misc.health.smithi050.stdout: 2024-09-11T14:42:53.192 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:42:52.445784+0000","last_up_change":"2024-09-11T14:42:51.439869+0000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:42:52.401770+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":586496889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":586496889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":586496889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":586496889}]},"public_addr":"172.21.15.50:6807/586496889","cluster_addr":"172.21.15.50:6806/586496889","heartbeat_back_addr":"172.21.15.50:6811/586496889","heartbeat_front_addr":"172.21.15.50:6814/586496889","state":["exists","up"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":916791714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":916791714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":916791714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":916791714}]},"public_addr":"172.21.15.50:6804/916791714","cluster_addr":"172.21.15.50:6803/916791714","heartbeat_back_addr":"172.21.15.50:6802/916791714","heartbeat_front_addr":"172.21.15.50:6805/916791714","state":["exists","up"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":704540503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":704540503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":704540503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":704540503}]},"public_addr":"172.21.15.50:6809/704540503","cluster_addr":"172.21.15.50:6808/704540503","heartbeat_back_addr":"172.21.15.50:6812/704540503","heartbeat_front_addr":"172.21.15.50:6813/704540503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:42:53.205 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T14:42:53.205 INFO:tasks.ceph:Creating RBD pool 2024-09-11T14:42:53.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T14:42:53.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:53.316+0000 7f6196b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:53.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:53.329+0000 7f6196b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.459 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2024-09-11T14:42:54.477 DEBUG:teuthology.orchestra.run.smithi050:> rbd --cluster ceph pool init rbd 2024-09-11T14:42:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:54.566+0000 7fb5ba40eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:54.567+0000 7fb5ba40eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:54.573+0000 7fb5ba40eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:57.607 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T14:42:57.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T14:42:57.608 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T14:42:57.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:57.717+0000 7f1e57791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:57.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:57.726+0000 7f1e57791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:57.925 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:57.938 INFO:teuthology.orchestra.run.smithi050.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-11T14:42:57.938 INFO:tasks.ceph_manager:config epoch is 1 2024-09-11T14:42:57.939 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T14:42:57.939 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T14:42:57.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T14:42:58.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.046+0000 7f436e04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.054+0000 7f436e04f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.304 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:58.316 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":718064291},{"type":"v1","addr":"172.21.15.50:6801","nonce":718064291}]},"active_addr":"172.21.15.50:6801/718064291","active_change":"2024-09-11T14:42:48.374423+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":709652317}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":4118840176}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2723053624}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1190586169}]}]} 2024-09-11T14:42:58.320 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T14:42:58.320 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T14:42:58.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:42:58.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.428+0000 7fd93cb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.436+0000 7fd93cb91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:58.632 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":15,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:42:57.525337+0000","last_up_change":"2024-09-11T14:42:51.439869+0000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:42:52.401770+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:42:53.570501+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":586496889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":586496889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":586496889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":586496889}]},"public_addr":"172.21.15.50:6807/586496889","cluster_addr":"172.21.15.50:6806/586496889","heartbeat_back_addr":"172.21.15.50:6811/586496889","heartbeat_front_addr":"172.21.15.50:6814/586496889","state":["exists","up"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":916791714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":916791714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":916791714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":916791714}]},"public_addr":"172.21.15.50:6804/916791714","cluster_addr":"172.21.15.50:6803/916791714","heartbeat_back_addr":"172.21.15.50:6802/916791714","heartbeat_front_addr":"172.21.15.50:6805/916791714","state":["exists","up"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":704540503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":704540503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":704540503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":704540503}]},"public_addr":"172.21.15.50:6809/704540503","cluster_addr":"172.21.15.50:6808/704540503","heartbeat_back_addr":"172.21.15.50:6812/704540503","heartbeat_front_addr":"172.21.15.50:6813/704540503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:42:58.645 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T14:42:58.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:42:58.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.752+0000 7fabfcb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:58.761+0000 7fabfcb51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:58.960 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:42:58.960 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":15,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:42:57.525337+0000","last_up_change":"2024-09-11T14:42:51.439869+0000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:42:52.401770+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:42:53.570501+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":586496889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":586496889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":586496889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":586496889}]},"public_addr":"172.21.15.50:6807/586496889","cluster_addr":"172.21.15.50:6806/586496889","heartbeat_back_addr":"172.21.15.50:6811/586496889","heartbeat_front_addr":"172.21.15.50:6814/586496889","state":["exists","up"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":916791714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":916791714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":916791714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":916791714}]},"public_addr":"172.21.15.50:6804/916791714","cluster_addr":"172.21.15.50:6803/916791714","heartbeat_back_addr":"172.21.15.50:6802/916791714","heartbeat_front_addr":"172.21.15.50:6805/916791714","state":["exists","up"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":704540503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":704540503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":704540503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":704540503}]},"public_addr":"172.21.15.50:6809/704540503","cluster_addr":"172.21.15.50:6808/704540503","heartbeat_back_addr":"172.21.15.50:6812/704540503","heartbeat_front_addr":"172.21.15.50:6813/704540503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:42:58.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T14:42:58.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T14:42:58.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T14:42:59.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.087+0000 7f50e0f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.087+0000 7f362f688640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.087+0000 7ffab14d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.097+0000 7ffab14d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.097+0000 7f50e0f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.097+0000 7f362f688640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.131 INFO:teuthology.orchestra.run.smithi050.stdout:38654705671 2024-09-11T14:42:59.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:42:59.133 INFO:teuthology.orchestra.run.smithi050.stdout:34359738376 2024-09-11T14:42:59.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:42:59.136 INFO:teuthology.orchestra.run.smithi050.stdout:38654705671 2024-09-11T14:42:59.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:42:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.240+0000 7f224cea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.251+0000 7f224cea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.287+0000 7f46a86ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.291+0000 7f8844663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.297+0000 7f46a86ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:42:59.302+0000 7f8844663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:42:59.458 INFO:teuthology.orchestra.run.smithi050.stdout:38654705668 2024-09-11T14:42:59.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-11T14:42:59.501 INFO:teuthology.orchestra.run.smithi050.stdout:38654705669 2024-09-11T14:42:59.510 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2024-09-11T14:42:59.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705669 for osd.2 2024-09-11T14:42:59.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-11T14:43:00.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T14:43:00.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T14:43:00.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T14:43:00.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.599+0000 7f3e8b246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.611+0000 7f3e8b246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.636+0000 7f72d4f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.648+0000 7f72d4f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.681+0000 7f30140d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:00.691+0000 7f30140d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:00.821 INFO:teuthology.orchestra.run.smithi050.stdout:38654705671 2024-09-11T14:43:00.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-11T14:43:00.835 DEBUG:teuthology.parallel:result is None 2024-09-11T14:43:00.856 INFO:teuthology.orchestra.run.smithi050.stdout:38654705671 2024-09-11T14:43:00.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.2 2024-09-11T14:43:00.869 DEBUG:teuthology.parallel:result is None 2024-09-11T14:43:00.898 INFO:teuthology.orchestra.run.smithi050.stdout:34359738376 2024-09-11T14:43:00.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-11T14:43:00.911 DEBUG:teuthology.parallel:result is None 2024-09-11T14:43:00.911 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:43:00.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:01.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.059+0000 7f2a00f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.067+0000 7f2a00f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.262 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:43:01.262 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:43:01.275 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:43:00.380429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.856571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.556201+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:42:57.556201+0000","last_peered":"2024-09-11T14:42:57.556201+0000","last_clean":"2024-09-11T14:42:57.556201+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:42:57.556201+0000","last_undegraded":"2024-09-11T14:42:57.556201+0000","last_fullsized":"2024-09-11T14:42:57.556201+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.567435+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:42:57.567435+0000","last_peered":"2024-09-11T14:42:57.567435+0000","last_clean":"2024-09-11T14:42:57.567435+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:42:57.567435+0000","last_undegraded":"2024-09-11T14:42:57.567435+0000","last_fullsized":"2024-09-11T14:42:57.567435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.560577+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:42:57.560577+0000","last_peered":"2024-09-11T14:42:57.560577+0000","last_clean":"2024-09-11T14:42:57.560577+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:42:57.560577+0000","last_undegraded":"2024-09-11T14:42:57.560577+0000","last_fullsized":"2024-09-11T14:42:57.560577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.559651+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:42:57.559651+0000","last_peered":"2024-09-11T14:42:57.559651+0000","last_clean":"2024-09-11T14:42:57.559651+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:42:57.559651+0000","last_undegraded":"2024-09-11T14:42:57.559651+0000","last_fullsized":"2024-09-11T14:42:57.559651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.580163+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:42:57.580163+0000","last_peered":"2024-09-11T14:42:57.580163+0000","last_clean":"2024-09-11T14:42:57.580163+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:42:57.580163+0000","last_undegraded":"2024-09-11T14:42:57.580163+0000","last_fullsized":"2024-09-11T14:42:57.580163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.572105+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:42:57.572105+0000","last_peered":"2024-09-11T14:42:57.572105+0000","last_clean":"2024-09-11T14:42:57.572105+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:42:57.572105+0000","last_undegraded":"2024-09-11T14:42:57.572105+0000","last_fullsized":"2024-09-11T14:42:57.572105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.563199+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:42:57.563199+0000","last_peered":"2024-09-11T14:42:57.563199+0000","last_clean":"2024-09-11T14:42:57.563199+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:42:57.563199+0000","last_undegraded":"2024-09-11T14:42:57.563199+0000","last_fullsized":"2024-09-11T14:42:57.563199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.554085+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:42:57.554085+0000","last_peered":"2024-09-11T14:42:57.554085+0000","last_clean":"2024-09-11T14:42:57.554085+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:42:57.554085+0000","last_undegraded":"2024-09-11T14:42:57.554085+0000","last_fullsized":"2024-09-11T14:42:57.554085+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.554674+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:42:57.554674+0000","last_peered":"2024-09-11T14:42:57.554674+0000","last_clean":"2024-09-11T14:42:57.554674+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:42:57.554674+0000","last_undegraded":"2024-09-11T14:42:57.554674+0000","last_fullsized":"2024-09-11T14:42:57.554674+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:01.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:43:01.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.382+0000 7fe80a711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.390+0000 7fe80a711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.584 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:43:01.585 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:43:01.597 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-11T14:43:00.380429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3372,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889812,"statfs":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.856571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.556201+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:42:57.556201+0000","last_peered":"2024-09-11T14:42:57.556201+0000","last_clean":"2024-09-11T14:42:57.556201+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:42:57.556201+0000","last_undegraded":"2024-09-11T14:42:57.556201+0000","last_fullsized":"2024-09-11T14:42:57.556201+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.567435+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:42:57.567435+0000","last_peered":"2024-09-11T14:42:57.567435+0000","last_clean":"2024-09-11T14:42:57.567435+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:42:57.567435+0000","last_undegraded":"2024-09-11T14:42:57.567435+0000","last_fullsized":"2024-09-11T14:42:57.567435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.560577+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:42:57.560577+0000","last_peered":"2024-09-11T14:42:57.560577+0000","last_clean":"2024-09-11T14:42:57.560577+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:42:57.560577+0000","last_undegraded":"2024-09-11T14:42:57.560577+0000","last_fullsized":"2024-09-11T14:42:57.560577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.559651+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:42:57.559651+0000","last_peered":"2024-09-11T14:42:57.559651+0000","last_clean":"2024-09-11T14:42:57.559651+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:42:57.559651+0000","last_undegraded":"2024-09-11T14:42:57.559651+0000","last_fullsized":"2024-09-11T14:42:57.559651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.580163+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:42:57.580163+0000","last_peered":"2024-09-11T14:42:57.580163+0000","last_clean":"2024-09-11T14:42:57.580163+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:42:57.580163+0000","last_undegraded":"2024-09-11T14:42:57.580163+0000","last_fullsized":"2024-09-11T14:42:57.580163+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.572105+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:42:57.572105+0000","last_peered":"2024-09-11T14:42:57.572105+0000","last_clean":"2024-09-11T14:42:57.572105+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:42:57.572105+0000","last_undegraded":"2024-09-11T14:42:57.572105+0000","last_fullsized":"2024-09-11T14:42:57.572105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.563199+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:42:57.563199+0000","last_peered":"2024-09-11T14:42:57.563199+0000","last_clean":"2024-09-11T14:42:57.563199+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:42:57.563199+0000","last_undegraded":"2024-09-11T14:42:57.563199+0000","last_fullsized":"2024-09-11T14:42:57.563199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.554085+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:42:57.554085+0000","last_peered":"2024-09-11T14:42:57.554085+0000","last_clean":"2024-09-11T14:42:57.554085+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:42:57.554085+0000","last_undegraded":"2024-09-11T14:42:57.554085+0000","last_fullsized":"2024-09-11T14:42:57.554085+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T14:42:57.554674+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:42:57.554674+0000","last_peered":"2024-09-11T14:42:57.554674+0000","last_clean":"2024-09-11T14:42:57.554674+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:42:57.554674+0000","last_undegraded":"2024-09-11T14:42:57.554674+0000","last_fullsized":"2024-09-11T14:42:57.554674+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655167488,"internally_reserved":0,"allocated":3452928,"data_stored":3452928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297292,"statfs":{"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1596,"kb_used_data":1596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296132,"statfs":{"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1340,"kb_used_data":1340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296388,"statfs":{"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551501312,"internally_reserved":0,"allocated":1372160,"data_stored":1372160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551239168,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552427008,"internally_reserved":0,"allocated":446464,"data_stored":446464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:43:01.598 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:43:01.598 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T14:43:01.598 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T14:43:01.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T14:43:01.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.705+0000 7fc9944d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:43:01.712+0000 7fc9944d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:43:01.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:43:01.945 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T14:43:01.958 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T14:43:01.959 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T14:43:01.978 INFO:tasks.workunit:Pulling workunits from ref dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:43:01.979 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T14:43:01.979 DEBUG:teuthology.orchestra.run.smithi050:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:43:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:43:01.997 INFO:teuthology.orchestra.run.smithi050.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T14:43:01.997 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:43:02.053 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T14:43:02.053 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T14:43:02.109 INFO:tasks.workunit:timeout=3h 2024-09-11T14:43:02.110 INFO:tasks.workunit:cleanup=True 2024-09-11T14:43:02.110 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/matan-b/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 2024-09-11T14:43:02.166 INFO:tasks.workunit.client.0.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T14:43:47.139 INFO:tasks.workunit.client.0.smithi050.stderr:Note: switching to 'dbebf1a3db6db7b5579464ee715edcf1b3e9cce2'. 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T14:43:47.140 INFO:tasks.workunit.client.0.smithi050.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: git switch -c 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr:Or undo this operation with: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: git switch - 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-09-11T14:43:47.141 INFO:tasks.workunit.client.0.smithi050.stderr:HEAD is now at dbebf1a3db6 qa/config/seastore: enable all subsystem debug 2024-09-11T14:43:47.143 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T14:43:47.204 INFO:tasks.workunit.client.0.smithi050.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T14:43:47.206 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:43:47.206 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T14:43:47.802 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T14:43:47.873 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T14:43:47.915 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T14:43:47.917 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:43:47.917 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T14:43:47.958 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T14:43:47.962 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-11T14:43:47.962 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T14:43:48.018 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-11T14:43:48.020 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-11T14:43:48.020 DEBUG:teuthology.orchestra.run.smithi050:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-11T14:43:48.096 INFO:tasks.workunit.client.0.smithi050.stdout:Running main() from gmock_main.cc 2024-09-11T14:43:48.097 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] Running 71 tests from 1 test suite. 2024-09-11T14:43:48.097 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment set-up. 2024-09-11T14:43:48.097 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 71 tests from TestClsRbd 2024-09-11T14:43:49.701 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-11T14:43:50.152 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_all_features (452 ms) 2024-09-11T14:43:50.152 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.copyup 2024-09-11T14:43:50.880 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.copyup (728 ms) 2024-09-11T14:43:50.880 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-11T14:43:51.101 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.sparse_copyup (222 ms) 2024-09-11T14:43:51.101 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-11T14:43:51.209 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_and_set_id (108 ms) 2024-09-11T14:43:51.210 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-11T14:43:51.414 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.add_remove_child (205 ms) 2024-09-11T14:43:51.414 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-11T14:43:51.950 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.directory_methods (536 ms) 2024-09-11T14:43:51.950 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.create 2024-09-11T14:43:52.231 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.create (280 ms) 2024-09-11T14:43:52.231 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_features 2024-09-11T14:43:52.263 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_features (32 ms) 2024-09-11T14:43:52.263 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-11T14:43:52.301 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_object_prefix (38 ms) 2024-09-11T14:43:52.301 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-11T14:43:52.330 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_create_timestamp (30 ms) 2024-09-11T14:43:52.331 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-11T14:43:52.359 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_access_timestamp (29 ms) 2024-09-11T14:43:52.359 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-11T14:43:52.389 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_modify_timestamp (29 ms) 2024-09-11T14:43:52.389 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-11T14:43:52.455 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_data_pool (67 ms) 2024-09-11T14:43:52.456 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.get_size 2024-09-11T14:43:52.555 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.get_size (100 ms) 2024-09-11T14:43:52.556 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.set_size 2024-09-11T14:43:52.648 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.set_size (93 ms) 2024-09-11T14:43:52.648 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-11T14:43:52.998 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.protection_status (350 ms) 2024-09-11T14:43:52.998 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-11T14:43:53.177 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.snapshot_limits (179 ms) 2024-09-11T14:43:53.178 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-11T14:43:54.190 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.parents_v1 (1012 ms) 2024-09-11T14:43:54.190 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-11T14:43:54.822 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.parents_v2 (633 ms) 2024-09-11T14:43:54.822 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-11T14:43:55.362 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.snapshots (539 ms) 2024-09-11T14:43:55.362 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-11T14:43:55.444 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.snapid_race (82 ms) 2024-09-11T14:43:55.444 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-11T14:43:55.601 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.stripingv2 (158 ms) 2024-09-11T14:43:55.602 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-11T14:43:55.631 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_save (30 ms) 2024-09-11T14:43:55.631 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-11T14:43:55.732 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_resize (101 ms) 2024-09-11T14:43:55.732 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-11T14:43:55.830 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_update (98 ms) 2024-09-11T14:43:55.830 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-11T14:43:55.835 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_load_enoent (5 ms) 2024-09-11T14:43:55.835 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-11T14:43:55.925 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_snap_add (90 ms) 2024-09-11T14:43:55.925 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-11T14:43:56.006 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.object_map_snap_remove (81 ms) 2024-09-11T14:43:56.006 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.flags 2024-09-11T14:43:56.112 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.flags (106 ms) 2024-09-11T14:43:56.113 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.metadata 2024-09-11T14:43:57.709 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.metadata (1596 ms) 2024-09-11T14:43:57.709 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.set_features 2024-09-11T14:43:57.837 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.set_features (128 ms) 2024-09-11T14:43:57.837 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror 2024-09-11T14:43:58.578 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror (742 ms) 2024-09-11T14:43:58.578 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-11T14:43:58.912 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror_image (333 ms) 2024-09-11T14:43:58.912 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-11T14:45:32.936 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror_image_status (94024 ms) 2024-09-11T14:45:32.937 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-11T14:45:35.621 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror_image_map (2685 ms) 2024-09-11T14:45:35.621 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-11T14:45:35.774 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror_instances (153 ms) 2024-09-11T14:45:35.774 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-11T14:45:36.016 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.mirror_snapshot (242 ms) 2024-09-11T14:45:36.016 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-11T14:45:36.063 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_dir_list (48 ms) 2024-09-11T14:45:36.063 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-11T14:45:36.108 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_dir_add (44 ms) 2024-09-11T14:45:36.108 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-11T14:45:36.163 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.dir_add_already_existing (55 ms) 2024-09-11T14:45:36.163 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-11T14:45:36.282 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_dir_rename (119 ms) 2024-09-11T14:45:36.282 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-11T14:45:36.351 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_dir_remove (69 ms) 2024-09-11T14:45:36.351 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-11T14:45:36.428 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (78 ms) 2024-09-11T14:45:36.428 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-11T14:45:36.472 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_image_add (43 ms) 2024-09-11T14:45:36.472 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-11T14:45:36.546 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_image_remove (75 ms) 2024-09-11T14:45:36.546 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-11T14:45:36.604 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_image_list (57 ms) 2024-09-11T14:45:36.604 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-11T14:45:36.694 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_image_clean (91 ms) 2024-09-11T14:45:36.695 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-11T14:45:36.752 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.image_group_add (58 ms) 2024-09-11T14:45:36.753 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-11T14:45:36.837 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.image_group_remove (84 ms) 2024-09-11T14:45:36.837 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-11T14:45:36.887 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.image_group_get (51 ms) 2024-09-11T14:45:36.887 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-11T14:45:36.921 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (34 ms) 2024-09-11T14:45:36.921 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-11T14:45:36.951 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (29 ms) 2024-09-11T14:45:36.951 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-11T14:45:37.006 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (55 ms) 2024-09-11T14:45:37.006 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-11T14:45:37.057 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (51 ms) 2024-09-11T14:45:37.058 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-11T14:45:37.103 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_set (45 ms) 2024-09-11T14:45:37.103 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-11T14:45:37.251 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_list (148 ms) 2024-09-11T14:45:37.251 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-11T14:45:37.640 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (389 ms) 2024-09-11T14:45:37.640 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-11T14:45:41.867 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (4227 ms) 2024-09-11T14:45:41.867 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-11T14:45:41.938 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_remove (72 ms) 2024-09-11T14:45:41.938 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_remove_without_order 2024-09-11T14:45:42.027 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_remove_without_order (89 ms) 2024-09-11T14:45:42.027 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-11T14:45:42.075 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (47 ms) 2024-09-11T14:45:42.075 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-11T14:45:42.223 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.trash_methods (149 ms) 2024-09-11T14:45:42.224 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.op_features 2024-09-11T14:45:42.383 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.op_features (160 ms) 2024-09-11T14:45:42.383 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-11T14:45:42.836 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.clone_parent (452 ms) 2024-09-11T14:45:42.836 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-11T14:45:43.087 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.clone_parent_ns (252 ms) 2024-09-11T14:45:43.087 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-11T14:45:43.370 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.clone_child (282 ms) 2024-09-11T14:45:43.370 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-11T14:45:43.496 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.namespace_methods (126 ms) 2024-09-11T14:45:43.496 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.migration 2024-09-11T14:45:43.740 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.migration (244 ms) 2024-09-11T14:45:43.740 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-11T14:45:43.923 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.migration_v1 (183 ms) 2024-09-11T14:45:43.923 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-11T14:45:45.764 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1842 ms) 2024-09-11T14:45:45.764 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-11T14:45:46.331 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsRbd.sparsify (566 ms) 2024-09-11T14:45:46.772 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 71 tests from TestClsRbd (116631 ms total) 2024-09-11T14:45:46.773 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-09-11T14:45:46.773 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment tear-down 2024-09-11T14:45:46.773 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] 71 tests from 1 test suite ran. (118676 ms total) 2024-09-11T14:45:46.773 INFO:tasks.workunit.client.0.smithi050.stdout:[ PASSED ] 71 tests. 2024-09-11T14:45:46.777 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T14:45:46.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T14:45:46.848 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2024-09-11T14:45:46.850 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2024-09-11T14:45:46.850 DEBUG:teuthology.orchestra.run.smithi050:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2024-09-11T14:45:46.950 INFO:tasks.workunit.client.0.smithi050.stdout:Running main() from gmock_main.cc 2024-09-11T14:45:46.950 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] Running 11 tests from 1 test suite. 2024-09-11T14:45:46.950 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment set-up. 2024-09-11T14:45:46.950 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 11 tests from ClsLock 2024-09-11T14:45:46.950 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestMultiLocking 2024-09-11T14:45:49.519 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:49.519 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: exclusive 2024-09-11T14:45:49.519 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:49.519 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:49.519 INFO:tasks.workunit.client.0.smithi050.stdout: client.4200 expiration=2024-09-11T14:47:49.410602+0000 addr=v1:172.21.15.50:0/1124312550 cookie= 2024-09-11T14:45:49.614 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:49.614 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: shared 2024-09-11T14:45:49.615 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:49.615 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:49.615 INFO:tasks.workunit.client.0.smithi050.stdout: client.4200 expiration=2024-09-11T14:47:49.589637+0000 addr=v1:172.21.15.50:0/1124312550 cookie= 2024-09-11T14:45:49.615 INFO:tasks.workunit.client.0.smithi050.stdout: client.4203 expiration=0.000000 addr=v1:172.21.15.50:0/3892754950 cookie= 2024-09-11T14:45:49.656 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:49.656 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: shared 2024-09-11T14:45:49.657 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:49.657 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:49.657 INFO:tasks.workunit.client.0.smithi050.stdout: client.4203 expiration=0.000000 addr=v1:172.21.15.50:0/3892754950 cookie= 2024-09-11T14:45:49.786 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestMultiLocking (2835 ms) 2024-09-11T14:45:49.786 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestMeta 2024-09-11T14:45:52.765 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:52.766 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: shared 2024-09-11T14:45:52.766 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:52.766 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:52.766 INFO:tasks.workunit.client.0.smithi050.stdout: client.4209 expiration=0.000000 addr=v1:172.21.15.50:0/933547368 cookie= 2024-09-11T14:45:52.850 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:52.850 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: exclusive 2024-09-11T14:45:52.850 INFO:tasks.workunit.client.0.smithi050.stdout: tag: new_tag 2024-09-11T14:45:52.850 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:52.850 INFO:tasks.workunit.client.0.smithi050.stdout: client.4206 expiration=0.000000 addr=v1:172.21.15.50:0/3596976144 cookie= 2024-09-11T14:45:53.801 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestMeta (4015 ms) 2024-09-11T14:45:53.801 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestCookie 2024-09-11T14:45:56.417 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:56.417 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: exclusive 2024-09-11T14:45:56.418 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:56.418 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:56.499 INFO:tasks.workunit.client.0.smithi050.stdout:lock: mylock 2024-09-11T14:45:56.500 INFO:tasks.workunit.client.0.smithi050.stdout: lock_type: shared 2024-09-11T14:45:56.500 INFO:tasks.workunit.client.0.smithi050.stdout: tag: 2024-09-11T14:45:56.500 INFO:tasks.workunit.client.0.smithi050.stdout: lockers: 2024-09-11T14:45:56.500 INFO:tasks.workunit.client.0.smithi050.stdout: client.4212 expiration=0.000000 addr=v1:172.21.15.50:0/2483303546 cookie= 2024-09-11T14:45:56.500 INFO:tasks.workunit.client.0.smithi050.stdout: client.4212 expiration=0.000000 addr=v1:172.21.15.50:0/2483303546 cookie=new cookie 2024-09-11T14:45:56.813 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestCookie (3013 ms) 2024-09-11T14:45:56.813 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestMultipleLocks 2024-09-11T14:45:59.823 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestMultipleLocks (3009 ms) 2024-09-11T14:45:59.823 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestLockDuration 2024-09-11T14:46:07.848 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestLockDuration (8025 ms) 2024-09-11T14:46:07.849 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestAssertLocked 2024-09-11T14:46:10.863 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestAssertLocked (3014 ms) 2024-09-11T14:46:10.863 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestSetCookie 2024-09-11T14:46:13.872 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestSetCookie (3010 ms) 2024-09-11T14:46:13.872 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestRenew 2024-09-11T14:46:34.904 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestRenew (21031 ms) 2024-09-11T14:46:34.904 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2024-09-11T14:46:41.925 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7022 ms) 2024-09-11T14:46:41.926 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2024-09-11T14:46:50.489 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8563 ms) 2024-09-11T14:46:50.489 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2024-09-11T14:46:58.510 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8022 ms) 2024-09-11T14:46:58.511 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 11 tests from ClsLock (71560 ms total) 2024-09-11T14:46:58.511 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-09-11T14:46:58.511 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment tear-down 2024-09-11T14:46:58.511 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] 11 tests from 1 test suite ran. (71561 ms total) 2024-09-11T14:46:58.511 INFO:tasks.workunit.client.0.smithi050.stdout:[ PASSED ] 11 tests. 2024-09-11T14:46:58.514 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T14:46:58.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T14:46:58.548 INFO:tasks.workunit:Running workunits matching cls/test_cls_journal.sh on client.0... 2024-09-11T14:46:58.550 INFO:tasks.workunit:Running workunit cls/test_cls_journal.sh... 2024-09-11T14:46:58.551 DEBUG:teuthology.orchestra.run.smithi050:workunit test cls/test_cls_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dbebf1a3db6db7b5579464ee715edcf1b3e9cce2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh 2024-09-11T14:46:58.637 INFO:tasks.workunit.client.0.smithi050.stdout:Running main() from gmock_main.cc 2024-09-11T14:46:58.637 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] Running 27 tests from 1 test suite. 2024-09-11T14:46:58.637 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment set-up. 2024-09-11T14:46:58.637 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 27 tests from TestClsJournal 2024-09-11T14:47:00.517 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.Create 2024-09-11T14:47:01.168 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.Create (650 ms) 2024-09-11T14:47:01.168 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.MinimumSet 2024-09-11T14:47:01.279 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.MinimumSet (112 ms) 2024-09-11T14:47:01.280 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.MinimumSetStale 2024-09-11T14:47:01.479 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.MinimumSetStale (199 ms) 2024-09-11T14:47:01.479 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.MinimumSetOrderConstraint 2024-09-11T14:47:01.564 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.MinimumSetOrderConstraint (85 ms) 2024-09-11T14:47:01.564 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ActiveSet 2024-09-11T14:47:01.663 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ActiveSet (100 ms) 2024-09-11T14:47:01.663 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ActiveSetStale 2024-09-11T14:47:01.722 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ActiveSetStale (58 ms) 2024-09-11T14:47:01.722 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.CreateDuplicate 2024-09-11T14:47:01.759 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.CreateDuplicate (37 ms) 2024-09-11T14:47:01.759 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.GetClient 2024-09-11T14:47:01.813 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.GetClient (54 ms) 2024-09-11T14:47:01.813 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientRegister 2024-09-11T14:47:01.866 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientRegister (53 ms) 2024-09-11T14:47:01.866 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientRegisterDuplicate 2024-09-11T14:47:01.924 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientRegisterDuplicate (58 ms) 2024-09-11T14:47:01.925 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientUpdateData 2024-09-11T14:47:02.010 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientUpdateData (87 ms) 2024-09-11T14:47:02.010 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientUpdateState 2024-09-11T14:47:02.099 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientUpdateState (89 ms) 2024-09-11T14:47:02.099 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientUnregister 2024-09-11T14:47:02.164 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientUnregister (65 ms) 2024-09-11T14:47:02.164 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientUnregisterDNE 2024-09-11T14:47:02.242 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientUnregisterDNE (78 ms) 2024-09-11T14:47:02.242 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientUnregisterPruneTags 2024-09-11T14:47:04.936 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientUnregisterPruneTags (2693 ms) 2024-09-11T14:47:04.936 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientCommit 2024-09-11T14:47:05.023 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientCommit (88 ms) 2024-09-11T14:47:05.023 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientCommitInvalid 2024-09-11T14:47:05.082 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientCommitInvalid (59 ms) 2024-09-11T14:47:05.082 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientCommitDNE 2024-09-11T14:47:05.095 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientCommitDNE (13 ms) 2024-09-11T14:47:05.095 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.ClientList 2024-09-11T14:47:07.914 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.ClientList (2819 ms) 2024-09-11T14:47:07.915 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.GetNextTagTid 2024-09-11T14:47:08.010 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.GetNextTagTid (96 ms) 2024-09-11T14:47:08.011 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.TagCreate 2024-09-11T14:47:08.206 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.TagCreate (196 ms) 2024-09-11T14:47:08.206 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.TagCreatePrunesTags 2024-09-11T14:47:08.383 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.TagCreatePrunesTags (176 ms) 2024-09-11T14:47:08.383 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.TagList 2024-09-11T14:47:11.414 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.TagList (3032 ms) 2024-09-11T14:47:11.414 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.GuardAppend 2024-09-11T14:47:11.446 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.GuardAppend (32 ms) 2024-09-11T14:47:11.446 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.GuardAppendDNE 2024-09-11T14:47:11.452 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.GuardAppendDNE (6 ms) 2024-09-11T14:47:11.452 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.GuardAppendOverflow 2024-09-11T14:47:11.484 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.GuardAppendOverflow (32 ms) 2024-09-11T14:47:11.484 INFO:tasks.workunit.client.0.smithi050.stdout:[ RUN ] TestClsJournal.Append 2024-09-11T14:47:11.639 INFO:tasks.workunit.client.0.smithi050.stdout:padding is not enabled 2024-09-11T14:47:11.639 INFO:tasks.workunit.client.0.smithi050.stdout:[ OK ] TestClsJournal.Append (155 ms) 2024-09-11T14:47:12.535 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] 27 tests from TestClsJournal (11122 ms total) 2024-09-11T14:47:12.535 INFO:tasks.workunit.client.0.smithi050.stdout: 2024-09-11T14:47:12.535 INFO:tasks.workunit.client.0.smithi050.stdout:[----------] Global test environment tear-down 2024-09-11T14:47:12.535 INFO:tasks.workunit.client.0.smithi050.stdout:[==========] 27 tests from 1 test suite ran. (13899 ms total) 2024-09-11T14:47:12.535 INFO:tasks.workunit.client.0.smithi050.stdout:[ PASSED ] 27 tests. 2024-09-11T14:47:12.540 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T14:47:12.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T14:47:12.573 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh'] on client.0... 2024-09-11T14:47:12.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T14:47:13.014 DEBUG:teuthology.parallel:result is None 2024-09-11T14:47:13.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:47:13.049 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:47:13.049 DEBUG:teuthology.orchestra.run.smithi050:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T14:47:13.116 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T14:47:13.116 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T14:47:13.126 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T14:47:13.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.277+0000 7ff475311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.288+0000 7ff475311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.488 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:13.488 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:47:13.502 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-11T14:47:12.529381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":14716,"kb_used_data":14716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156878468,"statfs":{"total":160658620416,"available":160643551232,"internally_reserved":0,"allocated":15069184,"data_stored":15069184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-17,"num_read_kb":0,"num_write":-32,"num_write_kb":-32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.098882"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.559207+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:47:01.559207+0000","last_peered":"2024-09-11T14:47:01.559207+0000","last_clean":"2024-09-11T14:47:01.559207+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:47:01.559207+0000","last_undegraded":"2024-09-11T14:47:01.559207+0000","last_fullsized":"2024-09-11T14:47:01.559207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.556506+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:47:01.556506+0000","last_peered":"2024-09-11T14:47:01.556506+0000","last_clean":"2024-09-11T14:47:01.556506+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:47:01.556506+0000","last_undegraded":"2024-09-11T14:47:01.556506+0000","last_fullsized":"2024-09-11T14:47:01.556506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.547995+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:47:01.547995+0000","last_peered":"2024-09-11T14:47:01.547995+0000","last_clean":"2024-09-11T14:47:01.547995+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:47:01.547995+0000","last_undegraded":"2024-09-11T14:47:01.547995+0000","last_fullsized":"2024-09-11T14:47:01.547995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.552007+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:47:01.552007+0000","last_peered":"2024-09-11T14:47:01.552007+0000","last_clean":"2024-09-11T14:47:01.552007+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:47:01.552007+0000","last_undegraded":"2024-09-11T14:47:01.552007+0000","last_fullsized":"2024-09-11T14:47:01.552007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":164,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.574287+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:47:01.574287+0000","last_peered":"2024-09-11T14:47:01.574287+0000","last_clean":"2024-09-11T14:47:01.574287+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:47:01.574287+0000","last_undegraded":"2024-09-11T14:47:01.574287+0000","last_fullsized":"2024-09-11T14:47:01.574287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.620991+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:47:01.620991+0000","last_peered":"2024-09-11T14:47:01.620991+0000","last_clean":"2024-09-11T14:47:01.620991+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:47:01.620991+0000","last_undegraded":"2024-09-11T14:47:01.620991+0000","last_fullsized":"2024-09-11T14:47:01.620991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.617296+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:47:01.617296+0000","last_peered":"2024-09-11T14:47:01.617296+0000","last_clean":"2024-09-11T14:47:01.617296+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:47:01.617296+0000","last_undegraded":"2024-09-11T14:47:01.617296+0000","last_fullsized":"2024-09-11T14:47:01.617296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.555473+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:47:01.555473+0000","last_peered":"2024-09-11T14:47:01.555473+0000","last_clean":"2024-09-11T14:47:01.555473+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:47:01.555473+0000","last_undegraded":"2024-09-11T14:47:01.555473+0000","last_fullsized":"2024-09-11T14:47:01.555473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":234,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.558757+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:47:01.558757+0000","last_peered":"2024-09-11T14:47:01.558757+0000","last_clean":"2024-09-11T14:47:01.558757+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:47:01.558757+0000","last_undegraded":"2024-09-11T14:47:01.558757+0000","last_fullsized":"2024-09-11T14:47:01.558757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160643203072,"internally_reserved":0,"allocated":15417344,"data_stored":15417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4676,"kb_used_data":4676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293052,"statfs":{"total":53552873472,"available":53548085248,"internally_reserved":0,"allocated":4788224,"data_stored":4788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4956,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292772,"statfs":{"total":53552873472,"available":53547798528,"internally_reserved":0,"allocated":5074944,"data_stored":5074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5084,"kb_used_data":5084,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292644,"statfs":{"total":53552873472,"available":53547667456,"internally_reserved":0,"allocated":5206016,"data_stored":5206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53547528192,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547945984,"internally_reserved":0,"allocated":4927488,"data_stored":4927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:13.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:13.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.617+0000 7f07daf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.626+0000 7f07daf11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.831 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:13.831 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:47:13.845 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-11T14:47:12.529381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":14716,"kb_used_data":14716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156878468,"statfs":{"total":160658620416,"available":160643551232,"internally_reserved":0,"allocated":15069184,"data_stored":15069184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-17,"num_read_kb":0,"num_write":-32,"num_write_kb":-32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.098882"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.559207+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:47:01.559207+0000","last_peered":"2024-09-11T14:47:01.559207+0000","last_clean":"2024-09-11T14:47:01.559207+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:47:01.559207+0000","last_undegraded":"2024-09-11T14:47:01.559207+0000","last_fullsized":"2024-09-11T14:47:01.559207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.556506+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:47:01.556506+0000","last_peered":"2024-09-11T14:47:01.556506+0000","last_clean":"2024-09-11T14:47:01.556506+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:47:01.556506+0000","last_undegraded":"2024-09-11T14:47:01.556506+0000","last_fullsized":"2024-09-11T14:47:01.556506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.547995+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:47:01.547995+0000","last_peered":"2024-09-11T14:47:01.547995+0000","last_clean":"2024-09-11T14:47:01.547995+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:47:01.547995+0000","last_undegraded":"2024-09-11T14:47:01.547995+0000","last_fullsized":"2024-09-11T14:47:01.547995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.552007+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:47:01.552007+0000","last_peered":"2024-09-11T14:47:01.552007+0000","last_clean":"2024-09-11T14:47:01.552007+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:47:01.552007+0000","last_undegraded":"2024-09-11T14:47:01.552007+0000","last_fullsized":"2024-09-11T14:47:01.552007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":164,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.574287+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:47:01.574287+0000","last_peered":"2024-09-11T14:47:01.574287+0000","last_clean":"2024-09-11T14:47:01.574287+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:47:01.574287+0000","last_undegraded":"2024-09-11T14:47:01.574287+0000","last_fullsized":"2024-09-11T14:47:01.574287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.620991+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:47:01.620991+0000","last_peered":"2024-09-11T14:47:01.620991+0000","last_clean":"2024-09-11T14:47:01.620991+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:47:01.620991+0000","last_undegraded":"2024-09-11T14:47:01.620991+0000","last_fullsized":"2024-09-11T14:47:01.620991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.617296+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:47:01.617296+0000","last_peered":"2024-09-11T14:47:01.617296+0000","last_clean":"2024-09-11T14:47:01.617296+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:47:01.617296+0000","last_undegraded":"2024-09-11T14:47:01.617296+0000","last_fullsized":"2024-09-11T14:47:01.617296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.555473+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:47:01.555473+0000","last_peered":"2024-09-11T14:47:01.555473+0000","last_clean":"2024-09-11T14:47:01.555473+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:47:01.555473+0000","last_undegraded":"2024-09-11T14:47:01.555473+0000","last_fullsized":"2024-09-11T14:47:01.555473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":234,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.558757+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:47:01.558757+0000","last_peered":"2024-09-11T14:47:01.558757+0000","last_clean":"2024-09-11T14:47:01.558757+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:47:01.558757+0000","last_undegraded":"2024-09-11T14:47:01.558757+0000","last_fullsized":"2024-09-11T14:47:01.558757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160643203072,"internally_reserved":0,"allocated":15417344,"data_stored":15417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4676,"kb_used_data":4676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293052,"statfs":{"total":53552873472,"available":53548085248,"internally_reserved":0,"allocated":4788224,"data_stored":4788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4956,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292772,"statfs":{"total":53552873472,"available":53547798528,"internally_reserved":0,"allocated":5074944,"data_stored":5074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5084,"kb_used_data":5084,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292644,"statfs":{"total":53552873472,"available":53547667456,"internally_reserved":0,"allocated":5206016,"data_stored":5206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53547528192,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547945984,"internally_reserved":0,"allocated":4927488,"data_stored":4927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:13.845 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T14:47:13.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:13.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.954+0000 7f9d1b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:13.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:13.962+0000 7f9d1b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.155 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:14.156 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:47:14.169 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-11T14:47:12.529381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":14716,"kb_used_data":14716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156878468,"statfs":{"total":160658620416,"available":160643551232,"internally_reserved":0,"allocated":15069184,"data_stored":15069184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-17,"num_read_kb":0,"num_write":-32,"num_write_kb":-32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.098882"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.559207+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:47:01.559207+0000","last_peered":"2024-09-11T14:47:01.559207+0000","last_clean":"2024-09-11T14:47:01.559207+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:47:01.559207+0000","last_undegraded":"2024-09-11T14:47:01.559207+0000","last_fullsized":"2024-09-11T14:47:01.559207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.556506+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:47:01.556506+0000","last_peered":"2024-09-11T14:47:01.556506+0000","last_clean":"2024-09-11T14:47:01.556506+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:47:01.556506+0000","last_undegraded":"2024-09-11T14:47:01.556506+0000","last_fullsized":"2024-09-11T14:47:01.556506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.547995+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:47:01.547995+0000","last_peered":"2024-09-11T14:47:01.547995+0000","last_clean":"2024-09-11T14:47:01.547995+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:47:01.547995+0000","last_undegraded":"2024-09-11T14:47:01.547995+0000","last_fullsized":"2024-09-11T14:47:01.547995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.552007+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:47:01.552007+0000","last_peered":"2024-09-11T14:47:01.552007+0000","last_clean":"2024-09-11T14:47:01.552007+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:47:01.552007+0000","last_undegraded":"2024-09-11T14:47:01.552007+0000","last_fullsized":"2024-09-11T14:47:01.552007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":164,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.574287+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:47:01.574287+0000","last_peered":"2024-09-11T14:47:01.574287+0000","last_clean":"2024-09-11T14:47:01.574287+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:47:01.574287+0000","last_undegraded":"2024-09-11T14:47:01.574287+0000","last_fullsized":"2024-09-11T14:47:01.574287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.620991+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:47:01.620991+0000","last_peered":"2024-09-11T14:47:01.620991+0000","last_clean":"2024-09-11T14:47:01.620991+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:47:01.620991+0000","last_undegraded":"2024-09-11T14:47:01.620991+0000","last_fullsized":"2024-09-11T14:47:01.620991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.617296+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:47:01.617296+0000","last_peered":"2024-09-11T14:47:01.617296+0000","last_clean":"2024-09-11T14:47:01.617296+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:47:01.617296+0000","last_undegraded":"2024-09-11T14:47:01.617296+0000","last_fullsized":"2024-09-11T14:47:01.617296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.555473+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:47:01.555473+0000","last_peered":"2024-09-11T14:47:01.555473+0000","last_clean":"2024-09-11T14:47:01.555473+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:47:01.555473+0000","last_undegraded":"2024-09-11T14:47:01.555473+0000","last_fullsized":"2024-09-11T14:47:01.555473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":234,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.558757+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:47:01.558757+0000","last_peered":"2024-09-11T14:47:01.558757+0000","last_clean":"2024-09-11T14:47:01.558757+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:47:01.558757+0000","last_undegraded":"2024-09-11T14:47:01.558757+0000","last_fullsized":"2024-09-11T14:47:01.558757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160643203072,"internally_reserved":0,"allocated":15417344,"data_stored":15417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4676,"kb_used_data":4676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293052,"statfs":{"total":53552873472,"available":53548085248,"internally_reserved":0,"allocated":4788224,"data_stored":4788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4956,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292772,"statfs":{"total":53552873472,"available":53547798528,"internally_reserved":0,"allocated":5074944,"data_stored":5074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5084,"kb_used_data":5084,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292644,"statfs":{"total":53552873472,"available":53547667456,"internally_reserved":0,"allocated":5206016,"data_stored":5206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53547528192,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547945984,"internally_reserved":0,"allocated":4927488,"data_stored":4927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:14.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T14:47:14.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:14.277+0000 7f3026f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:14.284+0000 7f3026f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:14.487 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:14.487 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":65,"fsid":"85b31bab-bc01-45a8-9735-fc6a813abd91","created":"2024-09-11T14:42:43.678836+0000","modified":"2024-09-11T14:47:12.525828+0000","last_up_change":"2024-09-11T14:42:51.439869+0000","last_in_change":"2024-09-11T14:42:45.593752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T14:42:52.401770+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T14:42:53.570501+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5469008b-2fe0-4f1f-9eb0-6ea59e2bab88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":586496889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":586496889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":586496889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":586496889}]},"public_addr":"172.21.15.50:6807/586496889","cluster_addr":"172.21.15.50:6806/586496889","heartbeat_back_addr":"172.21.15.50:6811/586496889","heartbeat_front_addr":"172.21.15.50:6814/586496889","state":["exists","up"]},{"osd":1,"uuid":"88f3e561-4b0e-47fd-b4ac-552fe6bcbea3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":916791714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":916791714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":916791714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":916791714}]},"public_addr":"172.21.15.50:6804/916791714","cluster_addr":"172.21.15.50:6803/916791714","heartbeat_back_addr":"172.21.15.50:6802/916791714","heartbeat_front_addr":"172.21.15.50:6805/916791714","state":["exists","up"]},{"osd":2,"uuid":"31867ae1-289b-464a-a7b8-455848503331","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":704540503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":704540503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":704540503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":704540503}]},"public_addr":"172.21.15.50:6809/704540503","cluster_addr":"172.21.15.50:6808/704540503","heartbeat_back_addr":"172.21.15.50:6812/704540503","heartbeat_front_addr":"172.21.15.50:6813/704540503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]},{"pool":3,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T14:47:15.502 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T14:47:15.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:47:15.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:15.624+0000 7fab92191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:15.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:15.638+0000 7fab92191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:15.656 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-11T14:47:15.656 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "" 2024-09-11T14:47:15.656 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-11T14:47:15.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T14:47:15.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:15.776+0000 7f1e43d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:15.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:15.786+0000 7f1e43d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:15.982 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T14:47:16.003 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T14:47:16.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:47:16.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.110+0000 7f6ef2711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.123+0000 7f6ef2711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.140 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-11T14:47:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "" 2024-09-11T14:47:16.140 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-11T14:47:16.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T14:47:16.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.258+0000 7f8a20f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.268+0000 7f8a20f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.464 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T14:47:16.480 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T14:47:16.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T14:47:16.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.594+0000 7f629e063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.608+0000 7f629e063640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.625 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-11T14:47:16.625 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "" 2024-09-11T14:47:16.625 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-11T14:47:16.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T14:47:16.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.744+0000 7f8c9b148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:16.753+0000 7f8c9b148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:16.950 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T14:47:16.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:17.076+0000 7f5e7a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:17.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:17.084+0000 7f5e7a391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:17.278 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:17.279 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:47:17.291 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-11T14:47:16.433374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":14716,"kb_used_data":14716,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156878468,"statfs":{"total":160658620416,"available":160643551232,"internally_reserved":0,"allocated":15069184,"data_stored":15069184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-17,"num_read_kb":0,"num_write":-32,"num_write_kb":-32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002105"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.559207+0000","last_change":"2024-09-11T14:42:57.570583+0000","last_active":"2024-09-11T14:47:01.559207+0000","last_peered":"2024-09-11T14:47:01.559207+0000","last_clean":"2024-09-11T14:47:01.559207+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:47:01.559207+0000","last_undegraded":"2024-09-11T14:47:01.559207+0000","last_fullsized":"2024-09-11T14:47:01.559207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.556506+0000","last_change":"2024-09-11T14:42:57.571184+0000","last_active":"2024-09-11T14:47:01.556506+0000","last_peered":"2024-09-11T14:47:01.556506+0000","last_clean":"2024-09-11T14:47:01.556506+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:47:01.556506+0000","last_undegraded":"2024-09-11T14:47:01.556506+0000","last_fullsized":"2024-09-11T14:47:01.556506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.547995+0000","last_change":"2024-09-11T14:42:57.573290+0000","last_active":"2024-09-11T14:47:01.547995+0000","last_peered":"2024-09-11T14:47:01.547995+0000","last_clean":"2024-09-11T14:47:01.547995+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:47:01.547995+0000","last_undegraded":"2024-09-11T14:47:01.547995+0000","last_fullsized":"2024-09-11T14:47:01.547995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.552007+0000","last_change":"2024-09-11T14:42:57.572198+0000","last_active":"2024-09-11T14:47:01.552007+0000","last_peered":"2024-09-11T14:47:01.552007+0000","last_clean":"2024-09-11T14:47:01.552007+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:47:01.552007+0000","last_undegraded":"2024-09-11T14:47:01.552007+0000","last_fullsized":"2024-09-11T14:47:01.552007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":164,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.574287+0000","last_change":"2024-09-11T14:42:57.580163+0000","last_active":"2024-09-11T14:47:01.574287+0000","last_peered":"2024-09-11T14:47:01.574287+0000","last_clean":"2024-09-11T14:47:01.574287+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:47:01.574287+0000","last_undegraded":"2024-09-11T14:47:01.574287+0000","last_fullsized":"2024-09-11T14:47:01.574287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.620991+0000","last_change":"2024-09-11T14:42:57.577914+0000","last_active":"2024-09-11T14:47:01.620991+0000","last_peered":"2024-09-11T14:47:01.620991+0000","last_clean":"2024-09-11T14:47:01.620991+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:47:01.620991+0000","last_undegraded":"2024-09-11T14:47:01.620991+0000","last_fullsized":"2024-09-11T14:47:01.620991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.617296+0000","last_change":"2024-09-11T14:42:57.570049+0000","last_active":"2024-09-11T14:47:01.617296+0000","last_peered":"2024-09-11T14:47:01.617296+0000","last_clean":"2024-09-11T14:47:01.617296+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:47:01.617296+0000","last_undegraded":"2024-09-11T14:47:01.617296+0000","last_fullsized":"2024-09-11T14:47:01.617296+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":163,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.555473+0000","last_change":"2024-09-11T14:42:57.572600+0000","last_active":"2024-09-11T14:47:01.555473+0000","last_peered":"2024-09-11T14:47:01.555473+0000","last_clean":"2024-09-11T14:47:01.555473+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:47:01.555473+0000","last_undegraded":"2024-09-11T14:47:01.555473+0000","last_fullsized":"2024-09-11T14:47:01.555473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:54.454946+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":234,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-11T14:47:01.558757+0000","last_change":"2024-09-11T14:42:53.583774+0000","last_active":"2024-09-11T14:47:01.558757+0000","last_peered":"2024-09-11T14:47:01.558757+0000","last_clean":"2024-09-11T14:47:01.558757+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:47:01.558757+0000","last_undegraded":"2024-09-11T14:47:01.558757+0000","last_fullsized":"2024-09-11T14:47:01.558757+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:42:52.445784+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160643203072,"internally_reserved":0,"allocated":15417344,"data_stored":15417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4676,"kb_used_data":4676,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293052,"statfs":{"total":53552873472,"available":53548085248,"internally_reserved":0,"allocated":4788224,"data_stored":4788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4956,"kb_used_data":4956,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292772,"statfs":{"total":53552873472,"available":53547798528,"internally_reserved":0,"allocated":5074944,"data_stored":5074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":5084,"kb_used_data":5084,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292644,"statfs":{"total":53552873472,"available":53547667456,"internally_reserved":0,"allocated":5206016,"data_stored":5206016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53547528192,"internally_reserved":0,"allocated":5345280,"data_stored":5345280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53547728896,"internally_reserved":0,"allocated":5144576,"data_stored":5144576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53547945984,"internally_reserved":0,"allocated":4927488,"data_stored":4927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:17.295 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.295 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.295 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.295 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T14:42:54.454946+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=54, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T14:42:52.445784+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=42, tm_sec=52, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=14, tm_min=47, tm_sec=14, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T14:47:17.296 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T14:47:37.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T14:47:37.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:37.420+0000 7f3707c88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:37.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:37.428+0000 7f3707c88640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-11T14:47:37.626 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-11T14:47:37.639 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-11T14:47:36.437989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":4992,"kb_used_data":4992,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156888192,"statfs":{"total":160658620416,"available":160653508608,"internally_reserved":0,"allocated":5111808,"data_stored":5111808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002833"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.475080+0000","last_change":"2024-09-11T14:47:16.520315+0000","last_active":"2024-09-11T14:47:16.475080+0000","last_peered":"2024-09-11T14:47:16.475080+0000","last_clean":"2024-09-11T14:47:16.475080+0000","last_became_active":"2024-09-11T14:42:55.584328+0000","last_became_peered":"2024-09-11T14:42:55.584328+0000","last_unstale":"2024-09-11T14:47:16.475080+0000","last_undegraded":"2024-09-11T14:47:16.475080+0000","last_fullsized":"2024-09-11T14:47:16.475080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.520132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.520132+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.474215+0000","last_change":"2024-09-11T14:47:16.501534+0000","last_active":"2024-09-11T14:47:16.474215+0000","last_peered":"2024-09-11T14:47:16.474215+0000","last_clean":"2024-09-11T14:47:16.474215+0000","last_became_active":"2024-09-11T14:42:55.648911+0000","last_became_peered":"2024-09-11T14:42:55.648911+0000","last_unstale":"2024-09-11T14:47:16.474215+0000","last_undegraded":"2024-09-11T14:47:16.474215+0000","last_fullsized":"2024-09-11T14:47:16.474215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.501209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.501209+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.471669+0000","last_change":"2024-09-11T14:47:16.526907+0000","last_active":"2024-09-11T14:47:16.471669+0000","last_peered":"2024-09-11T14:47:16.471669+0000","last_clean":"2024-09-11T14:47:16.471669+0000","last_became_active":"2024-09-11T14:42:55.569531+0000","last_became_peered":"2024-09-11T14:42:55.569531+0000","last_unstale":"2024-09-11T14:47:16.471669+0000","last_undegraded":"2024-09-11T14:47:16.471669+0000","last_fullsized":"2024-09-11T14:47:16.471669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.526706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.526706+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.469290+0000","last_change":"2024-09-11T14:47:16.528414+0000","last_active":"2024-09-11T14:47:16.469290+0000","last_peered":"2024-09-11T14:47:16.469290+0000","last_clean":"2024-09-11T14:47:16.469290+0000","last_became_active":"2024-09-11T14:42:55.600219+0000","last_became_peered":"2024-09-11T14:42:55.600219+0000","last_unstale":"2024-09-11T14:47:16.469290+0000","last_undegraded":"2024-09-11T14:47:16.469290+0000","last_fullsized":"2024-09-11T14:47:16.469290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.528238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.528238+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":169,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:15.984455+0000","last_change":"2024-09-11T14:47:16.005319+0000","last_active":"2024-09-11T14:47:15.984455+0000","last_peered":"2024-09-11T14:47:15.984455+0000","last_clean":"2024-09-11T14:47:15.984455+0000","last_became_active":"2024-09-11T14:42:55.593222+0000","last_became_peered":"2024-09-11T14:42:55.593222+0000","last_unstale":"2024-09-11T14:47:15.984455+0000","last_undegraded":"2024-09-11T14:47:15.984455+0000","last_fullsized":"2024-09-11T14:47:15.984455+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2024-09-11T14:47:16.005128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.005128+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.954548+0000","last_change":"2024-09-11T14:47:16.968858+0000","last_active":"2024-09-11T14:47:16.954548+0000","last_peered":"2024-09-11T14:47:16.954548+0000","last_clean":"2024-09-11T14:47:16.954548+0000","last_became_active":"2024-09-11T14:42:55.627206+0000","last_became_peered":"2024-09-11T14:42:55.627206+0000","last_unstale":"2024-09-11T14:47:16.954548+0000","last_undegraded":"2024-09-11T14:47:16.954548+0000","last_fullsized":"2024-09-11T14:47:16.954548+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.968657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.968657+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":167,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.951651+0000","last_change":"2024-09-11T14:47:16.969122+0000","last_active":"2024-09-11T14:47:16.951651+0000","last_peered":"2024-09-11T14:47:16.951651+0000","last_clean":"2024-09-11T14:47:16.951651+0000","last_became_active":"2024-09-11T14:42:55.602737+0000","last_became_peered":"2024-09-11T14:42:55.602737+0000","last_unstale":"2024-09-11T14:47:16.951651+0000","last_undegraded":"2024-09-11T14:47:16.951651+0000","last_fullsized":"2024-09-11T14:47:16.951651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T14:47:16.968955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.968955+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":168,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.467939+0000","last_change":"2024-09-11T14:47:16.541663+0000","last_active":"2024-09-11T14:47:16.467939+0000","last_peered":"2024-09-11T14:47:16.467939+0000","last_clean":"2024-09-11T14:47:16.467939+0000","last_became_active":"2024-09-11T14:42:55.555939+0000","last_became_peered":"2024-09-11T14:42:55.555939+0000","last_unstale":"2024-09-11T14:47:16.467939+0000","last_undegraded":"2024-09-11T14:47:16.467939+0000","last_fullsized":"2024-09-11T14:47:16.467939+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'1","last_scrub_stamp":"2024-09-11T14:47:16.541469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.541469+0000","last_clean_scrub_stamp":"2024-09-11T14:42:54.454946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":239,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-11T14:47:16.465484+0000","last_change":"2024-09-11T14:47:16.554758+0000","last_active":"2024-09-11T14:47:16.465484+0000","last_peered":"2024-09-11T14:47:16.465484+0000","last_clean":"2024-09-11T14:47:16.465484+0000","last_became_active":"2024-09-11T14:42:53.560056+0000","last_became_peered":"2024-09-11T14:42:53.560056+0000","last_unstale":"2024-09-11T14:47:16.465484+0000","last_undegraded":"2024-09-11T14:47:16.465484+0000","last_fullsized":"2024-09-11T14:47:16.465484+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-09-11T14:47:16.554532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T14:47:16.554532+0000","last_clean_scrub_stamp":"2024-09-11T14:42:52.445784+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160653508608,"internally_reserved":0,"allocated":5111808,"data_stored":5111808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53550669824,"internally_reserved":0,"allocated":2203648,"data_stored":2203648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":960,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296768,"statfs":{"total":53552873472,"available":53551890432,"internally_reserved":0,"allocated":983040,"data_stored":983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2152,"kb_used_data":2152,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295576,"statfs":{"total":53552873472,"available":53550669824,"internally_reserved":0,"allocated":2203648,"data_stored":2203648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1880,"kb_used_data":1880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52295848,"statfs":{"total":53552873472,"available":53550948352,"internally_reserved":0,"allocated":1925120,"data_stored":1925120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53550669824,"internally_reserved":0,"allocated":2203648,"data_stored":2203648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53550948352,"internally_reserved":0,"allocated":1925120,"data_stored":1925120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550669824,"internally_reserved":0,"allocated":2203648,"data_stored":2203648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551890432,"internally_reserved":0,"allocated":983040,"data_stored":983040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T14:47:37.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T14:47:37.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:37.744+0000 7f6740663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-11T14:47:37.756+0000 7f6740663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-11T14:47:37.976 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T14:47:37.977 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T14:47:37.977 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T14:47:37.977 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:38.060 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T14:47:38.060 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T14:47:38.060 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:38.129 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T14:47:38.129 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T14:47:38.129 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:38.214 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T14:47:38.215 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T14:47:38.215 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T14:47:38.215 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:38.245 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T14:47:38.245 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T14:47:38.245 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T14:47:38.245 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T14:47:38.297 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T14:47:38.297 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T14:47:38.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-11T14:47:38.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-11T14:47:38.365 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:47:38.365 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T14:47:38.572 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:47:38.572 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T14:47:38.686 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:47:38.687 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T14:47:38.744 INFO:tasks.ceph:Archiving mon data... 2024-09-11T14:47:38.745 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373/data/mon.a.tgz 2024-09-11T14:47:38.746 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-11T14:47:38.761 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.egF2oUMvfu 2024-09-11T14:47:38.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.egF2oUMvfu 2024-09-11T14:47:38.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.egF2oUMvfu 2024-09-11T14:47:38.965 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.egF2oUMvfu is 233KB 2024-09-11T14:47:39.033 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.egF2oUMvfu 2024-09-11T14:47:39.048 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T14:47:39.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T14:47:39.170 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T14:47:39.170 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T14:47:39.171 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373/remote/smithi050/crash 2024-09-11T14:47:39.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T14:47:39.235 INFO:tasks.ceph:Compressing logs... 2024-09-11T14:47:39.235 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T14:47:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T14:47:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29271.log 2024-09-11T14:47:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29358.log 2024-09-11T14:47:39.311 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29271.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29398.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29358.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28297.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28535.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29398.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29398.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30236.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28297.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29504.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28535.log.gz 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-11T14:47:39.312 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30236.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30386.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29504.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30470.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30386.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30386.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29106.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30470.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29106.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27349.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29106.log.gz 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30764.log 2024-09-11T14:47:39.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.27349.log: gzip -5 --verbose -- 67.7% /var/log/ceph/ceph-client.admin.30512.log -- replaced with /var/log/ceph/ceph.tmp-client.admin.27349.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28744.log 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30764.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30512.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30512.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28495.log 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30428.log 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30638.log 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28495.log.gz 2024-09-11T14:47:39.314 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30428.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30428.log.gz 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30848.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30638.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30638.log.gz 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30848.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.30848.log.gz -- 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr: /var/log/ceph/ceph-client.admin.29064.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29064.log: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29064.log.gz 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28581.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29190.log 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29484.log: /var/log/ceph/ceph-client.admin.28581.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-09-11T14:47:39.315 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29912.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28581.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30344.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29190.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29912.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29912.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29652.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30344.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30680.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29652.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29652.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29524.log 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30680.log.gz 2024-09-11T14:47:39.316 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29013.log 2024-09-11T14:47:39.317 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.29524.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29524.log.gz /var/log/ceph/ceph.log 2024-09-11T14:47:39.317 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29013.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.29013.log.gz 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30554.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28969.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30596.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30554.log.gz 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29272.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28969.log.gz 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30596.log.gz 2024-09-11T14:47:39.319 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30722.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29272.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29272.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28385.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.30722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28774.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30722.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29694.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28385.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28913.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28774.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28774.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29610.log 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29694.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.28913.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28913.log.gz 2024-09-11T14:47:39.320 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.29610.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29610.log.gz 2024-09-11T14:47:39.341 INFO:teuthology.orchestra.run.smithi050.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T14:47:39.449 INFO:teuthology.orchestra.run.smithi050.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T14:47:40.467 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T14:47:40.964 INFO:teuthology.orchestra.run.smithi050.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T14:47:41.508 INFO:teuthology.orchestra.run.smithi050.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T14:47:41.509 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-11T14:47:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m2.219s 2024-09-11T14:47:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m5.029s 2024-09-11T14:47:41.510 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.212s 2024-09-11T14:47:41.510 INFO:tasks.ceph:Archiving logs... 2024-09-11T14:47:41.510 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373/remote/smithi050/log 2024-09-11T14:47:41.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T14:47:42.609 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T14:47:42.617 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T14:47:42.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T14:47:42.670 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T14:47:42.678 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T14:47:42.678 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T14:47:42.699 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-11T14:47:42.703 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T14:47:42.704 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-11T14:47:42.704 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 18 +873us[ +863us] +/- 54ms 2024-09-11T14:47:42.704 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -627us[ -627us] +/- 53ms 2024-09-11T14:47:42.704 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -532us[ -532us] +/- 84ms 2024-09-11T14:47:42.704 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T14:47:42.704 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T14:47:42.712 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T14:47:42.713 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T14:47:42.739 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T14:47:42.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-11T14:47:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:42.836 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T14:47:42.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T14:47:42.873 INFO:teuthology.task.internal:Duration was 703.760496 seconds 2024-09-11T14:47:42.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T14:47:42.906 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T14:47:42.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T14:47:42.947 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T14:47:43.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-11T14:47:43.353 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T14:47:43.353 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-09-11T14:47:43.354 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T14:47:43.416 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T14:47:43.416 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T14:47:43.484 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T14:47:43.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T14:47:43.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T14:47:43.639 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T14:47:43.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T14:47:43.664 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T14:47:43.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:43.728 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-09-11T14:47:43.742 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T14:47:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T14:47:43.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T14:47:43.811 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T14:47:43.811 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-09-11_13:56:22-crimson-rados-main-distro-crimson-smithi/7901373/remote/smithi050 2024-09-11T14:47:43.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T14:47:43.877 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T14:47:43.877 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T14:47:43.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T14:47:43.940 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T14:47:43.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T14:47:43.966 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T14:47:43.966 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T14:47:43.982 INFO:teuthology.orchestra.run.smithi050.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 14:47 /home/ubuntu/cephtest 2024-09-11T14:47:43.983 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T14:47:44.000 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T14:47:44.069 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rbd_cls_tests} duration: 703.7604961395264 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-09-11T14:47:44.069 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T14:47:44.154 INFO:teuthology.run:pass