2024-09-14T05:00:47.560 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-14T05:00:47.575 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T05:00:47.652 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293 branch: main description: rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/recovery conf/{disable-pool-app}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900293' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: rbd_persistent_cache_mode: rwl rbd_persistent_cache_path: /home/ubuntu/cephtest/rbd-pwl-cache rbd_persistent_cache_size: 1073741824 rbd_plugins: pwl_cache global: mon warn on pool no app: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 workunit: branch: main sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - - mon.b - mgr.y - osd.2 - osd.3 - client.0 seed: 5839 sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 sleep_before_teardown: 0 subset: 99/128 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a targets: smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8kpNW67uWt3jLixrDmx6tIjTrAhFQ/hooIyMI/wJjjLjQY7XzlaRUtT+gerQCqEaIlEYMYxNpqryTIaa12wx0= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTKoicBGnMGzZsKnv74ZXTdyUe9z6sNsHJOrkGDG1aIKopy0wOIH/fz8ch0qoUc+068+kzqoIOkZhlbkBvzxc0= tasks: - install: null - ceph: null - exec: client.0: - mkdir -m 777 /home/ubuntu/cephtest/rbd-pwl-cache - exec_on_cleanup: client.0: - rm -rf /home/ubuntu/cephtest/rbd-pwl-cache - rbd.create_image: client.0: image_format: 2 image_name: testimage image_size: 10240 - rbd_pwl_cache_recovery: client.0: image_name: testimage teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_20:16:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-14T05:00:47.652 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa; will attempt to use it 2024-09-14T05:00:47.800 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa/tasks 2024-09-14T05:00:47.800 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-14T05:00:47.801 INFO:teuthology.task.internal:Checking packages... 2024-09-14T05:00:47.822 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7f24e589328ae9e639c0009d93340e5b235d0a47' 2024-09-14T05:00:47.822 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T05:00:47.822 INFO:teuthology.packaging:ref: None 2024-09-14T05:00:47.822 INFO:teuthology.packaging:tag: None 2024-09-14T05:00:47.822 INFO:teuthology.packaging:branch: main 2024-09-14T05:00:47.822 INFO:teuthology.packaging:sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:00:47.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-14T05:00:48.053 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4907-ga2782315-1jammy 2024-09-14T05:00:48.054 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-14T05:00:48.061 INFO:teuthology.task.internal:no buildpackages task found 2024-09-14T05:00:48.061 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-14T05:00:48.088 INFO:teuthology.task.internal:Saving configuration 2024-09-14T05:00:48.099 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-14T05:00:48.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-14T05:00:48.132 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 04:55:11.183759', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOSj5we3vwLW0u41IM59qU3BeXoHfxiudi3It0afPmMS'} 2024-09-14T05:00:48.152 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 04:55:11.182180', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzmAqwPFRh31mTX5pgVMHJLds3r279sbuVrfsGIO8xv'} 2024-09-14T05:00:48.153 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-14T05:00:48.159 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1'] 2024-09-14T05:00:48.159 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.2', 'osd.3', 'client.0'] 2024-09-14T05:00:48.159 INFO:teuthology.run_tasks:Running task console_log... 2024-09-14T05:00:48.217 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa3e46ab010>, signals=[15]) 2024-09-14T05:00:48.217 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-14T05:00:48.223 INFO:teuthology.task.internal:Opening connections... 2024-09-14T05:00:48.223 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:00:48.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:00:48.309 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:00:48.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:00:48.390 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-14T05:00:48.396 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-09-14T05:00:48.401 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-09-14T05:00:48.401 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-09-14T05:00:48.446 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-14T05:00:48.446 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Ubuntu" 2024-09-14T05:00:48.446 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="22.04" 2024-09-14T05:00:48.446 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_CODENAME=jammy 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:ID=ubuntu 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE=debian 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-14T05:00:48.447 INFO:teuthology.orchestra.run.smithi144.stdout:UBUNTU_CODENAME=jammy 2024-09-14T05:00:48.448 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-09-14T05:00:48.466 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-09-14T05:00:48.470 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-09-14T05:00:48.471 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-09-14T05:00:48.515 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-14T05:00:48.515 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="Ubuntu" 2024-09-14T05:00:48.515 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="22.04" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_CODENAME=jammy 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:ID=ubuntu 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE=debian 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-14T05:00:48.516 INFO:teuthology.orchestra.run.smithi194.stdout:UBUNTU_CODENAME=jammy 2024-09-14T05:00:48.516 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-09-14T05:00:48.540 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-14T05:00:48.550 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-14T05:00:48.556 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-14T05:00:48.556 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-09-14T05:00:48.559 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-09-14T05:00:48.562 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-14T05:00:48.568 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-14T05:00:48.568 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-09-14T05:00:48.608 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-09-14T05:00:48.624 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-14T05:00:48.676 INFO:teuthology.run_tasks:Running task kernel... 2024-09-14T05:00:48.690 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-14T05:00:48.690 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-14T05:00:48.690 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T05:00:48.690 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T05:00:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:00:48.695 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-14T05:00:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:00:48.696 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-14T05:00:48.742 INFO:teuthology.orchestra.run.smithi144.stdout:5.15.0-56-generic 2024-09-14T05:00:48.743 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.15.0-56-generic 2024-09-14T05:00:48.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get clean 2024-09-14T05:00:48.744 INFO:teuthology.orchestra.run.smithi194.stdout:5.15.0-56-generic 2024-09-14T05:00:48.745 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.15.0-56-generic 2024-09-14T05:00:48.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get clean 2024-09-14T05:00:48.889 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get update 2024-09-14T05:00:48.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-14T05:00:49.055 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:00:49.055 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-14T05:00:49.055 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-14T05:00:49.115 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-14T05:00:49.179 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:00:49.179 INFO:teuthology.orchestra.run.smithi194.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-14T05:00:49.227 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-14T05:00:49.281 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-14T05:00:49.353 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-14T05:00:49.382 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-14T05:00:49.386 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-14T05:00:49.388 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-14T05:00:49.388 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-14T05:00:49.388 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-14T05:00:49.409 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-14T05:00:49.411 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-14T05:00:49.415 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-14T05:00:49.417 INFO:teuthology.orchestra.run.smithi144.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-14T05:00:49.421 INFO:teuthology.orchestra.run.smithi144.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-14T05:00:49.425 INFO:teuthology.orchestra.run.smithi144.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-14T05:00:49.427 INFO:teuthology.orchestra.run.smithi144.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-14T05:00:49.440 INFO:teuthology.orchestra.run.smithi144.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-14T05:00:49.440 INFO:teuthology.orchestra.run.smithi144.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-14T05:00:49.441 INFO:teuthology.orchestra.run.smithi144.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-14T05:00:49.441 INFO:teuthology.orchestra.run.smithi144.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-14T05:00:49.455 INFO:teuthology.orchestra.run.smithi144.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-14T05:00:49.458 INFO:teuthology.orchestra.run.smithi144.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-14T05:00:49.459 INFO:teuthology.orchestra.run.smithi144.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-14T05:00:49.478 INFO:teuthology.orchestra.run.smithi144.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-14T05:00:49.478 INFO:teuthology.orchestra.run.smithi144.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-14T05:00:49.482 INFO:teuthology.orchestra.run.smithi144.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-14T05:00:49.482 INFO:teuthology.orchestra.run.smithi144.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-14T05:00:49.492 INFO:teuthology.orchestra.run.smithi144.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-14T05:00:49.498 INFO:teuthology.orchestra.run.smithi144.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-14T05:00:49.500 INFO:teuthology.orchestra.run.smithi144.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-14T05:00:49.500 INFO:teuthology.orchestra.run.smithi144.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-14T05:00:49.501 INFO:teuthology.orchestra.run.smithi144.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-14T05:00:49.511 INFO:teuthology.orchestra.run.smithi144.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-14T05:00:49.512 INFO:teuthology.orchestra.run.smithi144.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-14T05:00:49.512 INFO:teuthology.orchestra.run.smithi144.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-14T05:00:49.512 INFO:teuthology.orchestra.run.smithi144.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-14T05:00:49.518 INFO:teuthology.orchestra.run.smithi144.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-14T05:00:49.518 INFO:teuthology.orchestra.run.smithi144.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-14T05:00:49.519 INFO:teuthology.orchestra.run.smithi144.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-14T05:00:49.519 INFO:teuthology.orchestra.run.smithi144.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-14T05:00:49.519 INFO:teuthology.orchestra.run.smithi144.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-14T05:00:49.528 INFO:teuthology.orchestra.run.smithi144.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-14T05:00:49.584 INFO:teuthology.orchestra.run.smithi194.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-14T05:00:49.599 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-14T05:00:49.722 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-14T05:00:49.927 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-14T05:00:50.020 INFO:teuthology.orchestra.run.smithi194.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-14T05:00:50.036 INFO:teuthology.orchestra.run.smithi194.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-14T05:00:50.045 INFO:teuthology.orchestra.run.smithi194.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-14T05:00:50.046 INFO:teuthology.orchestra.run.smithi194.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-14T05:00:50.047 INFO:teuthology.orchestra.run.smithi194.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-14T05:00:50.076 INFO:teuthology.orchestra.run.smithi194.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-14T05:00:50.090 INFO:teuthology.orchestra.run.smithi194.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-14T05:00:50.091 INFO:teuthology.orchestra.run.smithi194.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-14T05:00:50.092 INFO:teuthology.orchestra.run.smithi194.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-14T05:00:50.143 INFO:teuthology.orchestra.run.smithi194.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-14T05:00:50.213 INFO:teuthology.orchestra.run.smithi194.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-14T05:00:50.213 INFO:teuthology.orchestra.run.smithi194.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-14T05:00:50.213 INFO:teuthology.orchestra.run.smithi194.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-14T05:00:50.220 INFO:teuthology.orchestra.run.smithi194.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-14T05:00:50.220 INFO:teuthology.orchestra.run.smithi194.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-14T05:00:50.223 INFO:teuthology.orchestra.run.smithi194.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-14T05:00:50.239 INFO:teuthology.orchestra.run.smithi194.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-14T05:00:50.240 INFO:teuthology.orchestra.run.smithi194.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-14T05:00:50.242 INFO:teuthology.orchestra.run.smithi194.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-14T05:00:50.242 INFO:teuthology.orchestra.run.smithi194.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-14T05:00:50.242 INFO:teuthology.orchestra.run.smithi194.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-14T05:00:50.243 INFO:teuthology.orchestra.run.smithi194.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-14T05:00:50.252 INFO:teuthology.orchestra.run.smithi194.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-14T05:00:50.256 INFO:teuthology.orchestra.run.smithi194.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-14T05:00:50.257 INFO:teuthology.orchestra.run.smithi194.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-14T05:00:50.257 INFO:teuthology.orchestra.run.smithi194.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-14T05:00:50.258 INFO:teuthology.orchestra.run.smithi194.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-14T05:00:50.282 INFO:teuthology.orchestra.run.smithi194.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-14T05:00:50.323 INFO:teuthology.orchestra.run.smithi194.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-14T05:00:50.323 INFO:teuthology.orchestra.run.smithi194.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-14T05:00:50.324 INFO:teuthology.orchestra.run.smithi194.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-14T05:00:50.343 INFO:teuthology.orchestra.run.smithi194.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-14T05:00:50.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-14T05:00:50.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-14T05:00:50.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-14T05:00:50.345 INFO:teuthology.orchestra.run.smithi194.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-14T05:00:50.404 INFO:teuthology.orchestra.run.smithi194.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-14T05:00:55.727 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 16.0 MB in 2s (7,502 kB/s) 2024-09-14T05:00:56.239 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 16.0 MB in 3s (5,894 kB/s) 2024-09-14T05:00:57.173 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:00:57.190 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-14T05:00:57.254 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:00:57.463 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-14T05:00:57.464 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-14T05:00:57.650 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:00:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:00:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:00:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:00:57.651 INFO:teuthology.orchestra.run.smithi144.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-14T05:00:57.652 INFO:teuthology.orchestra.run.smithi144.stdout: python2.7-minimal 2024-09-14T05:00:57.652 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:00:57.653 INFO:teuthology.orchestra.run.smithi144.stdout:The following additional packages will be installed: 2024-09-14T05:00:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-14T05:00:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-14T05:00:57.653 INFO:teuthology.orchestra.run.smithi144.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-14T05:00:57.655 INFO:teuthology.orchestra.run.smithi144.stdout:Suggested packages: 2024-09-14T05:00:57.655 INFO:teuthology.orchestra.run.smithi144.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-14T05:00:57.655 INFO:teuthology.orchestra.run.smithi144.stdout:Recommended packages: 2024-09-14T05:00:57.655 INFO:teuthology.orchestra.run.smithi144.stdout: thermald 2024-09-14T05:00:57.702 INFO:teuthology.orchestra.run.smithi144.stdout:The following NEW packages will be installed: 2024-09-14T05:00:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-14T05:00:57.702 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-14T05:00:57.703 INFO:teuthology.orchestra.run.smithi144.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-14T05:00:57.704 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages will be upgraded: 2024-09-14T05:00:57.704 INFO:teuthology.orchestra.run.smithi144.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-14T05:00:57.715 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:00:57.735 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-14T05:00:57.756 INFO:teuthology.orchestra.run.smithi144.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-14T05:00:57.756 INFO:teuthology.orchestra.run.smithi144.stdout:Need to get 113 MB of archives. 2024-09-14T05:00:57.756 INFO:teuthology.orchestra.run.smithi144.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-14T05:00:57.756 INFO:teuthology.orchestra.run.smithi144.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-14T05:00:57.793 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:00:57.989 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2024-09-14T05:00:57.989 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2024-09-14T05:00:58.159 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:00:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:00:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:00:58.159 INFO:teuthology.orchestra.run.smithi194.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:00:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-14T05:00:58.160 INFO:teuthology.orchestra.run.smithi194.stdout: python2.7-minimal 2024-09-14T05:00:58.160 INFO:teuthology.orchestra.run.smithi194.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:00:58.160 INFO:teuthology.orchestra.run.smithi194.stdout:The following additional packages will be installed: 2024-09-14T05:00:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-14T05:00:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-14T05:00:58.161 INFO:teuthology.orchestra.run.smithi194.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-14T05:00:58.162 INFO:teuthology.orchestra.run.smithi194.stdout:Suggested packages: 2024-09-14T05:00:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-14T05:00:58.162 INFO:teuthology.orchestra.run.smithi194.stdout:Recommended packages: 2024-09-14T05:00:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: thermald 2024-09-14T05:00:58.188 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-14T05:00:58.205 INFO:teuthology.orchestra.run.smithi194.stdout:The following NEW packages will be installed: 2024-09-14T05:00:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-14T05:00:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-14T05:00:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-14T05:00:58.207 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages will be upgraded: 2024-09-14T05:00:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-14T05:00:58.258 INFO:teuthology.orchestra.run.smithi194.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-14T05:00:58.258 INFO:teuthology.orchestra.run.smithi194.stdout:Need to get 113 MB of archives. 2024-09-14T05:00:58.258 INFO:teuthology.orchestra.run.smithi194.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-14T05:00:58.258 INFO:teuthology.orchestra.run.smithi194.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-14T05:00:58.272 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-14T05:00:58.709 INFO:teuthology.orchestra.run.smithi194.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-14T05:00:58.817 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-14T05:00:59.037 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-14T05:00:59.037 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-14T05:00:59.037 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-14T05:00:59.154 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-14T05:00:59.175 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-14T05:00:59.638 INFO:teuthology.orchestra.run.smithi194.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-14T05:00:59.639 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-14T05:00:59.639 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-14T05:00:59.648 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 113 MB in 2s (74.1 MB/s) 2024-09-14T05:00:59.762 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-14T05:00:59.833 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-14T05:00:59.854 INFO:teuthology.orchestra.run.smithi194.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-14T05:01:00.283 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 113 MB in 2s (69.1 MB/s) 2024-09-14T05:01:00.450 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-14T05:01:01.876 INFO:teuthology.orchestra.run.smithi144.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-14T05:01:01.880 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:01.947 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:02.352 INFO:teuthology.orchestra.run.smithi194.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-14T05:01:02.357 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:02.538 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:03.998 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-14T05:01:04.019 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:04.108 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:04.576 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-14T05:01:04.597 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:04.634 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:04.722 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-14T05:01:04.743 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:04.829 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:05.267 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-14T05:01:05.287 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:05.325 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:11.639 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:11.763 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:11.835 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:11.966 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:12.087 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:12.233 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:12.290 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:12.386 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:12.528 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-14T05:01:12.550 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-14T05:01:12.603 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-14T05:01:12.697 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-14T05:01:12.720 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-14T05:01:12.764 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-14T05:01:18.313 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-14T05:01:18.350 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:18.379 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:18.454 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-14T05:01:18.491 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-14T05:01:18.529 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:21.106 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:21.179 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-14T05:01:21.233 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:21.322 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-14T05:01:21.567 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-14T05:01:21.655 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-14T05:01:21.689 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:21.776 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:21.815 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-14T05:01:21.902 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-14T05:01:21.942 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:22.030 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:23.248 INFO:teuthology.orchestra.run.smithi144.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:23.248 INFO:teuthology.orchestra.run.smithi144.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:23.248 INFO:teuthology.orchestra.run.smithi144.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:23.248 INFO:teuthology.orchestra.run.smithi144.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:23.380 INFO:teuthology.orchestra.run.smithi194.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:23.381 INFO:teuthology.orchestra.run.smithi194.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:23.381 INFO:teuthology.orchestra.run.smithi194.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:23.381 INFO:teuthology.orchestra.run.smithi194.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:23.395 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:23.533 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:25.347 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:25.509 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:26.748 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-14T05:01:26.874 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-14T05:01:26.910 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-14T05:01:27.000 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:27.036 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-14T05:01:27.076 INFO:teuthology.orchestra.run.smithi144.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-14T05:01:27.076 INFO:teuthology.orchestra.run.smithi144.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:27.162 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-14T05:01:27.228 INFO:teuthology.orchestra.run.smithi194.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-14T05:01:27.229 INFO:teuthology.orchestra.run.smithi194.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:41.417 INFO:teuthology.orchestra.run.smithi194.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-14T05:01:41.417 INFO:teuthology.orchestra.run.smithi194.stdout:Sourcing file `/etc/default/grub' 2024-09-14T05:01:41.455 INFO:teuthology.orchestra.run.smithi194.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-14T05:01:41.469 INFO:teuthology.orchestra.run.smithi194.stdout:Generating grub configuration file ... 2024-09-14T05:01:41.563 INFO:teuthology.orchestra.run.smithi144.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-14T05:01:41.563 INFO:teuthology.orchestra.run.smithi144.stdout:Sourcing file `/etc/default/grub' 2024-09-14T05:01:41.574 INFO:teuthology.orchestra.run.smithi144.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-14T05:01:41.590 INFO:teuthology.orchestra.run.smithi144.stdout:Generating grub configuration file ... 2024-09-14T05:01:41.695 INFO:teuthology.orchestra.run.smithi194.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:41.713 INFO:teuthology.orchestra.run.smithi194.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:41.884 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:41.904 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:42.006 INFO:teuthology.orchestra.run.smithi194.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:42.012 INFO:teuthology.orchestra.run.smithi194.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:42.075 INFO:teuthology.orchestra.run.smithi194.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-14T05:01:42.081 INFO:teuthology.orchestra.run.smithi194.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-14T05:01:42.177 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:42.183 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:42.253 INFO:teuthology.orchestra.run.smithi144.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-14T05:01:42.259 INFO:teuthology.orchestra.run.smithi144.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-14T05:01:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-14T05:01:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-14T05:01:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-14T05:01:42.335 INFO:teuthology.orchestra.run.smithi194.stdout:done 2024-09-14T05:01:42.434 INFO:teuthology.orchestra.run.smithi144.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-14T05:01:42.434 INFO:teuthology.orchestra.run.smithi144.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-14T05:01:42.435 INFO:teuthology.orchestra.run.smithi144.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-14T05:01:42.473 INFO:teuthology.orchestra.run.smithi144.stdout:done 2024-09-14T05:01:42.833 DEBUG:teuthology.orchestra.run.smithi194:> dpkg -s linux-image-generic 2024-09-14T05:01:42.857 INFO:teuthology.orchestra.run.smithi194.stdout:Package: linux-image-generic 2024-09-14T05:01:42.857 INFO:teuthology.orchestra.run.smithi194.stdout:Status: install ok installed 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Priority: optional 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Section: kernel 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Installed-Size: 21 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Maintainer: Ubuntu Kernel Team 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: amd64 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Source: linux-meta 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Version: 5.15.0.121.121 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Recommends: thermald 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout:Description: Generic Linux kernel image 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout: This package will always depend on the latest generic kernel image 2024-09-14T05:01:42.858 INFO:teuthology.orchestra.run.smithi194.stdout: available. 2024-09-14T05:01:42.859 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-14T05:01:42.859 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T05:01:42.859 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T05:01:42.859 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-14T05:01:42.859 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T05:01:42.859 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get clean 2024-09-14T05:01:42.876 DEBUG:teuthology.orchestra.run.smithi144:> dpkg -s linux-image-generic 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Package: linux-image-generic 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Status: install ok installed 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Priority: optional 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Section: kernel 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Installed-Size: 21 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Maintainer: Ubuntu Kernel Team 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: amd64 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Source: linux-meta 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Version: 5.15.0.121.121 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Recommends: thermald 2024-09-14T05:01:42.904 INFO:teuthology.orchestra.run.smithi144.stdout:Description: Generic Linux kernel image 2024-09-14T05:01:42.905 INFO:teuthology.orchestra.run.smithi144.stdout: This package will always depend on the latest generic kernel image 2024-09-14T05:01:42.905 INFO:teuthology.orchestra.run.smithi144.stdout: available. 2024-09-14T05:01:42.905 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-14T05:01:42.905 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T05:01:42.905 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T05:01:42.905 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-14T05:01:42.905 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T05:01:42.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get clean 2024-09-14T05:01:43.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get update 2024-09-14T05:01:43.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-14T05:01:43.171 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-14T05:01:43.256 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:01:43.256 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-14T05:01:43.272 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-14T05:01:43.302 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:01:43.359 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-14T05:01:43.373 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-14T05:01:43.454 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-14T05:01:44.652 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:01:44.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-14T05:01:44.703 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:01:44.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-14T05:01:44.731 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:01:44.775 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:01:44.938 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2024-09-14T05:01:44.939 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2024-09-14T05:01:44.961 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-14T05:01:44.962 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-14T05:01:45.123 INFO:teuthology.orchestra.run.smithi194.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-14T05:01:45.124 INFO:teuthology.orchestra.run.smithi194.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-14T05:01:45.125 INFO:teuthology.orchestra.run.smithi194.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:01:45.129 INFO:teuthology.orchestra.run.smithi144.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-14T05:01:45.129 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:01:45.129 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:01:45.129 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:01:45.129 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:01:45.130 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-14T05:01:45.130 INFO:teuthology.orchestra.run.smithi144.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-14T05:01:45.130 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:01:45.165 INFO:teuthology.orchestra.run.smithi194.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-14T05:01:45.167 DEBUG:teuthology.orchestra.run.smithi194:> dpkg -s linux-image-generic 2024-09-14T05:01:45.168 INFO:teuthology.orchestra.run.smithi144.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-14T05:01:45.170 DEBUG:teuthology.orchestra.run.smithi144:> dpkg -s linux-image-generic 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Package: linux-image-generic 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Status: install ok installed 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Priority: optional 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Section: kernel 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Installed-Size: 21 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Maintainer: Ubuntu Kernel Team 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: amd64 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Source: linux-meta 2024-09-14T05:01:45.179 INFO:teuthology.orchestra.run.smithi194.stdout:Version: 5.15.0.121.121 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout:Recommends: thermald 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout:Description: Generic Linux kernel image 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: This package will always depend on the latest generic kernel image 2024-09-14T05:01:45.180 INFO:teuthology.orchestra.run.smithi194.stdout: available. 2024-09-14T05:01:45.180 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-14T05:01:45.227 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.QHFl8antlc 2024-09-14T05:01:45.227 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QHFl8antlc 2024-09-14T05:01:45.228 INFO:teuthology.orchestra.run.smithi144.stdout:Package: linux-image-generic 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Status: install ok installed 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Priority: optional 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Section: kernel 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Installed-Size: 21 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Maintainer: Ubuntu Kernel Team 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: amd64 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Source: linux-meta 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Version: 5.15.0.121.121 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-14T05:01:45.229 INFO:teuthology.orchestra.run.smithi144.stdout:Recommends: thermald 2024-09-14T05:01:45.230 INFO:teuthology.orchestra.run.smithi144.stdout:Description: Generic Linux kernel image 2024-09-14T05:01:45.230 INFO:teuthology.orchestra.run.smithi144.stdout: This package will always depend on the latest generic kernel image 2024-09-14T05:01:45.230 INFO:teuthology.orchestra.run.smithi144.stdout: available. 2024-09-14T05:01:45.230 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-14T05:01:45.278 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.Ofq3BWpf19 2024-09-14T05:01:45.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Ofq3BWpf19 2024-09-14T05:01:45.283 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.QHFl8antlc 2024-09-14T05:01:45.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.Ofq3BWpf19 2024-09-14T05:01:45.394 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.QHFl8antlc is 10KB 2024-09-14T05:01:45.407 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.QHFl8antlc 2024-09-14T05:01:45.412 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.454 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.Ofq3BWpf19 is 10KB 2024-09-14T05:01:45.466 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:01:45.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.467 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.468 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.Ofq3BWpf19 2024-09-14T05:01:45.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.528 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:01:45.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.529 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-14T05:01:45.543 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-14T05:01:45.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo update-grub 2024-09-14T05:01:45.594 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-14T05:01:45.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo update-grub 2024-09-14T05:01:46.714 INFO:teuthology.orchestra.run.smithi194.stderr:Sourcing file `/etc/default/grub' 2024-09-14T05:01:46.715 INFO:teuthology.orchestra.run.smithi194.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-14T05:01:46.718 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-09-14T05:01:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:Sourcing file `/etc/default/grub' 2024-09-14T05:01:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-14T05:01:46.817 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-09-14T05:01:46.909 INFO:teuthology.orchestra.run.smithi194.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:46.914 INFO:teuthology.orchestra.run.smithi194.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:46.995 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-14T05:01:47.001 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-14T05:01:47.158 INFO:teuthology.orchestra.run.smithi194.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:47.163 INFO:teuthology.orchestra.run.smithi194.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:47.206 INFO:teuthology.orchestra.run.smithi194.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-14T05:01:47.211 INFO:teuthology.orchestra.run.smithi194.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-14T05:01:47.242 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-14T05:01:47.248 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-14T05:01:47.289 INFO:teuthology.orchestra.run.smithi144.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-14T05:01:47.295 INFO:teuthology.orchestra.run.smithi144.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-14T05:01:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-14T05:01:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-14T05:01:47.370 INFO:teuthology.orchestra.run.smithi194.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-14T05:01:47.382 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-09-14T05:01:47.400 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-09-14T05:01:47.448 INFO:teuthology.orchestra.run.smithi144.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-14T05:01:47.449 INFO:teuthology.orchestra.run.smithi144.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-14T05:01:47.449 INFO:teuthology.orchestra.run.smithi144.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-14T05:01:47.469 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-09-14T05:01:47.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-09-14T05:02:17.427 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-14T05:02:17.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-14T05:02:17.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:02:17.490 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-14T05:02:17.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:02:17.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:02:35.837 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-14T05:02:44.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:02:44.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:02:47.901 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-14T05:02:51.965 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-14T05:02:59.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:02:59.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:03:00.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-14T05:03:00.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:03:32.390 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-14T05:03:32.787 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-14T05:03:33.098 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:03:33.098 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-14T05:03:33.098 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-14T05:03:33.104 INFO:teuthology.orchestra.run.smithi144.stdout:5.15.0-121-generic 2024-09-14T05:03:33.104 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-14T05:03:33.104 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T05:03:33.105 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-14T05:03:33.626 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-14T05:03:33.626 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-14T05:03:33.626 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-14T05:03:33.675 INFO:teuthology.orchestra.run.smithi194.stdout:5.15.0-121-generic 2024-09-14T05:03:33.675 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-14T05:03:33.675 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T05:03:33.675 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-14T05:03:34.105 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-14T05:03:34.106 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T05:03:34.187 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-09-14T05:03:34.207 DEBUG:teuthology.parallel:result is None 2024-09-14T05:03:34.676 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-14T05:03:34.676 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T05:03:34.814 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-09-14T05:03:34.848 DEBUG:teuthology.parallel:result is None 2024-09-14T05:03:34.848 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-14T05:03:34.856 INFO:teuthology.task.internal:Creating test directory... 2024-09-14T05:03:34.856 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T05:03:34.859 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T05:03:34.864 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-14T05:03:34.870 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-14T05:03:34.877 INFO:teuthology.task.internal:Creating archive directory... 2024-09-14T05:03:34.877 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T05:03:34.907 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T05:03:34.933 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-14T05:03:34.943 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-14T05:03:34.944 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T05:03:34.967 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T05:03:34.983 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:03:34.990 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:03:34.993 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:03:35.002 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:03:35.004 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-14T05:03:35.018 INFO:teuthology.task.internal:Configuring sudo... 2024-09-14T05:03:35.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T05:03:35.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T05:03:35.066 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-14T05:03:35.080 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-14T05:03:35.081 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T05:03:35.107 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T05:03:35.115 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:03:35.163 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:03:35.211 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:03:35.211 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T05:03:35.283 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:03:35.289 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:03:35.342 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:03:35.342 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T05:03:35.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-09-14T05:03:35.407 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-09-14T05:03:35.544 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-14T05:03:35.553 INFO:teuthology.task.internal:Starting timer... 2024-09-14T05:03:35.553 INFO:teuthology.run_tasks:Running task pcp... 2024-09-14T05:03:35.564 INFO:teuthology.run_tasks:Running task selinux... 2024-09-14T05:03:35.573 DEBUG:teuthology.task.selinux:Excluding smithi144: OS 'ubuntu' does not support SELinux 2024-09-14T05:03:35.573 DEBUG:teuthology.task.selinux:Excluding smithi194: OS 'ubuntu' does not support SELinux 2024-09-14T05:03:35.573 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-14T05:03:35.573 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-14T05:03:35.573 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-14T05:03:35.573 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-14T05:03:35.581 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-14T05:03:35.585 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-14T05:03:35.714 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-14T05:03:35.730 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-14T05:03:35.731 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi144.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-14T05:11:01.469 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi144.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-09-14T05:11:01.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:11:01.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:11:01.555 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-14T05:11:01.628 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-14T05:11:01.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-14T05:11:01.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:11:01.711 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-14T05:11:01.789 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-14T05:11:01.789 INFO:teuthology.run_tasks:Running task clock... 2024-09-14T05:11:01.802 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-14T05:11:01.802 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T05:11:01.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:11:01.805 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T05:11:01.805 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Command line: ntpd -gq 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: ---------------------------------------------------- 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: ntp-4 is maintained by Network Time Foundation, 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: corporation. Support and training for ntp-4 are 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: available at https://www.nwtime.org/support 2024-09-14T05:11:01.833 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: ---------------------------------------------------- 2024-09-14T05:11:01.834 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: proto: precision = 0.061 usec (-24) 2024-09-14T05:11:01.834 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: basedate set to 2022-02-04 2024-09-14T05:11:01.834 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: gps base set to 2022-02-06 (week 2196) 2024-09-14T05:11:01.834 INFO:teuthology.orchestra.run.smithi144.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-14T05:11:01.835 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-14T05:11:01.835 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: restrict ::: KOD does nothing without LIMITED. 2024-09-14T05:11:01.835 INFO:teuthology.orchestra.run.smithi144.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen and drop on 0 v6wildcard [::]:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen normally on 2 lo 127.0.0.1:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen normally on 3 enp3s0f1 172.21.15.144:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen normally on 4 lo [::1]:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5f%5]:123 2024-09-14T05:11:01.836 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:01 ntpd[18491]: Listening on routing socket on fd #22 for interface updates 2024-09-14T05:11:01.860 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-14T05:11:01.860 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Command line: ntpd -gq 2024-09-14T05:11:01.860 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: ---------------------------------------------------- 2024-09-14T05:11:01.860 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: ntp-4 is maintained by Network Time Foundation, 2024-09-14T05:11:01.860 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-14T05:11:01.861 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: corporation. Support and training for ntp-4 are 2024-09-14T05:11:01.861 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: available at https://www.nwtime.org/support 2024-09-14T05:11:01.861 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: ---------------------------------------------------- 2024-09-14T05:11:01.861 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: proto: precision = 0.044 usec (-24) 2024-09-14T05:11:01.861 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: basedate set to 2022-02-04 2024-09-14T05:11:01.862 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: gps base set to 2022-02-06 (week 2196) 2024-09-14T05:11:01.862 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-14T05:11:01.862 INFO:teuthology.orchestra.run.smithi194.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-14T05:11:01.862 INFO:teuthology.orchestra.run.smithi194.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: restrict ::: KOD does nothing without LIMITED. 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen and drop on 0 v6wildcard [::]:123 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen normally on 2 lo 127.0.0.1:123 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen normally on 3 enp3s0f1 172.21.15.194:123 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen normally on 4 lo [::1]:123 2024-09-14T05:11:01.863 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea5%5]:123 2024-09-14T05:11:01.864 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:01 ntpd[18453]: Listening on routing socket on fd #22 for interface updates 2024-09-14T05:11:02.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:02.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:02.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:02.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:02.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:02.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:02.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:02.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:02 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:03.833 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:03.833 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:03.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:03.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:03.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:03.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:03 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:03.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:03 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:04.833 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:04.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:04 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:04 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:05.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:05 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:05.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:05.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:05 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:06.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:06 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:06.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:06 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:07.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:07 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:07.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:07.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:07 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:08.833 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:08.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:08 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:08.861 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:08.862 INFO:teuthology.orchestra.run.smithi194.stdout:14 Sep 05:11:08 ntpd[18453]: ntpd: time slew -0.000986 s 2024-09-14T05:11:08.862 INFO:teuthology.orchestra.run.smithi194.stdout:ntpd: time slew -0.000986s 2024-09-14T05:11:08.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-14T05:11:08.862 INFO:teuthology.orchestra.run.smithi194.stderr:14 Sep 05:11:08 ntpd[18453]: can't open /var/log/ntpstats/loopstats.20240914: Permission denied 2024-09-14T05:11:08.912 INFO:teuthology.orchestra.run.smithi194.stdout: remote refid st t when poll reach delay offset jitter 2024-09-14T05:11:08.912 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================== 2024-09-14T05:11:08.912 INFO:teuthology.orchestra.run.smithi194.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:08.912 INFO:teuthology.orchestra.run.smithi194.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:08.913 INFO:teuthology.orchestra.run.smithi194.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:08.913 INFO:teuthology.orchestra.run.smithi194.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: can't open /var/log/ntpstats/rawstats.20240914: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-14T05:11:09.834 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: can't open /var/log/ntpstats/peerstats.20240914: Permission denied 2024-09-14T05:11:09.835 INFO:teuthology.orchestra.run.smithi144.stdout:14 Sep 05:11:09 ntpd[18491]: ntpd: time slew -0.000827 s 2024-09-14T05:11:09.835 INFO:teuthology.orchestra.run.smithi144.stdout:ntpd: time slew -0.000827s 2024-09-14T05:11:09.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-14T05:11:09.835 INFO:teuthology.orchestra.run.smithi144.stderr:14 Sep 05:11:09 ntpd[18491]: can't open /var/log/ntpstats/loopstats.20240914: Permission denied 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout: remote refid st t when poll reach delay offset jitter 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================== 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:09.886 INFO:teuthology.orchestra.run.smithi144.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-14T05:11:09.887 INFO:teuthology.run_tasks:Running task install... 2024-09-14T05:11:09.895 DEBUG:teuthology.task.install:project ceph 2024-09-14T05:11:09.895 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}} 2024-09-14T05:11:09.896 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'} 2024-09-14T05:11:09.896 INFO:teuthology.task.install:Using flavor: default 2024-09-14T05:11:09.901 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-14T05:11:09.901 INFO:teuthology.task.install:extra packages: [] 2024-09-14T05:11:09.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-key list | grep Ceph 2024-09-14T05:11:09.902 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-key list | grep Ceph 2024-09-14T05:11:09.966 INFO:teuthology.orchestra.run.smithi194.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-14T05:11:09.989 INFO:teuthology.orchestra.run.smithi144.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-14T05:11:09.996 INFO:teuthology.orchestra.run.smithi194.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-14T05:11:09.996 INFO:teuthology.orchestra.run.smithi194.stdout:uid [ unknown] Ceph.com (release key) 2024-09-14T05:11:09.996 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-14T05:11:09.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:11:10.018 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-14T05:11:10.018 INFO:teuthology.orchestra.run.smithi144.stdout:uid [ unknown] Ceph.com (release key) 2024-09-14T05:11:10.019 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-14T05:11:10.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:11:10.146 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default/ 2024-09-14T05:11:10.146 INFO:teuthology.task.install.deb:Package version is 19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:11:10.159 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default/ 2024-09-14T05:11:10.159 INFO:teuthology.task.install.deb:Package version is 19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:11:10.256 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:11:10.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-14T05:11:10.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo apt-get update 2024-09-14T05:11:10.272 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:11:10.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-14T05:11:10.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo apt-get update 2024-09-14T05:11:10.436 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-14T05:11:10.459 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:11:10.470 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-14T05:11:10.487 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-14T05:11:10.489 INFO:teuthology.orchestra.run.smithi194.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy InRelease 2024-09-14T05:11:10.510 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-14T05:11:10.514 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy InRelease 2024-09-14T05:11:10.532 INFO:teuthology.orchestra.run.smithi194.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-14T05:11:10.537 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-14T05:11:10.553 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,420 B] 2024-09-14T05:11:10.559 INFO:teuthology.orchestra.run.smithi144.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-14T05:11:10.560 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-14T05:11:10.579 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-14T05:11:10.581 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-14T05:11:10.585 INFO:teuthology.orchestra.run.smithi144.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-14T05:11:10.606 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,420 B] 2024-09-14T05:11:10.642 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-14T05:11:10.720 INFO:teuthology.orchestra.run.smithi194.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-14T05:11:10.809 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 29.3 kB in 0s (87.2 kB/s) 2024-09-14T05:11:10.864 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 29.3 kB in 0s (71.1 kB/s) 2024-09-14T05:11:11.876 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:11:11.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4775-g7f24e589-1jammy cephadm=19.3.0-4775-g7f24e589-1jammy ceph-mds=19.3.0-4775-g7f24e589-1jammy ceph-mgr=19.3.0-4775-g7f24e589-1jammy ceph-common=19.3.0-4775-g7f24e589-1jammy ceph-fuse=19.3.0-4775-g7f24e589-1jammy ceph-test=19.3.0-4775-g7f24e589-1jammy ceph-volume=19.3.0-4775-g7f24e589-1jammy radosgw=19.3.0-4775-g7f24e589-1jammy python3-rados=19.3.0-4775-g7f24e589-1jammy python3-rgw=19.3.0-4775-g7f24e589-1jammy python3-cephfs=19.3.0-4775-g7f24e589-1jammy python3-rbd=19.3.0-4775-g7f24e589-1jammy libcephfs2=19.3.0-4775-g7f24e589-1jammy libcephfs-dev=19.3.0-4775-g7f24e589-1jammy librados2=19.3.0-4775-g7f24e589-1jammy librbd1=19.3.0-4775-g7f24e589-1jammy rbd-fuse=19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:11:11.933 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:11:11.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4775-g7f24e589-1jammy cephadm=19.3.0-4775-g7f24e589-1jammy ceph-mds=19.3.0-4775-g7f24e589-1jammy ceph-mgr=19.3.0-4775-g7f24e589-1jammy ceph-common=19.3.0-4775-g7f24e589-1jammy ceph-fuse=19.3.0-4775-g7f24e589-1jammy ceph-test=19.3.0-4775-g7f24e589-1jammy ceph-volume=19.3.0-4775-g7f24e589-1jammy radosgw=19.3.0-4775-g7f24e589-1jammy python3-rados=19.3.0-4775-g7f24e589-1jammy python3-rgw=19.3.0-4775-g7f24e589-1jammy python3-cephfs=19.3.0-4775-g7f24e589-1jammy python3-rbd=19.3.0-4775-g7f24e589-1jammy libcephfs2=19.3.0-4775-g7f24e589-1jammy libcephfs-dev=19.3.0-4775-g7f24e589-1jammy librados2=19.3.0-4775-g7f24e589-1jammy librbd1=19.3.0-4775-g7f24e589-1jammy rbd-fuse=19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:11:11.956 INFO:teuthology.orchestra.run.smithi144.stdout:Reading package lists... 2024-09-14T05:11:12.004 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists... 2024-09-14T05:11:12.165 INFO:teuthology.orchestra.run.smithi144.stdout:Building dependency tree... 2024-09-14T05:11:12.166 INFO:teuthology.orchestra.run.smithi144.stdout:Reading state information... 2024-09-14T05:11:12.190 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree... 2024-09-14T05:11:12.190 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information... 2024-09-14T05:11:12.373 INFO:teuthology.orchestra.run.smithi144.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:11:12.373 INFO:teuthology.orchestra.run.smithi144.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:11:12.373 INFO:teuthology.orchestra.run.smithi144.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:11:12.374 INFO:teuthology.orchestra.run.smithi144.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:11:12.374 INFO:teuthology.orchestra.run.smithi144.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-14T05:11:12.374 INFO:teuthology.orchestra.run.smithi144.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-14T05:11:12.374 INFO:teuthology.orchestra.run.smithi144.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:11:12.374 INFO:teuthology.orchestra.run.smithi144.stdout:The following additional packages will be installed: 2024-09-14T05:11:12.375 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-14T05:11:12.375 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-14T05:11:12.375 INFO:teuthology.orchestra.run.smithi144.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-14T05:11:12.375 INFO:teuthology.orchestra.run.smithi144.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-14T05:11:12.375 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-14T05:11:12.377 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages were automatically installed and are no longer required: 2024-09-14T05:11:12.377 INFO:teuthology.orchestra.run.smithi194.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout:The following additional packages will be installed: 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi144.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi144.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi144.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-14T05:11:12.378 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-14T05:11:12.379 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet zip 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout:Suggested packages: 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout:Recommended packages: 2024-09-14T05:11:12.380 INFO:teuthology.orchestra.run.smithi144.stdout: btrfs-tools 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-14T05:11:12.382 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet zip 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout:Suggested packages: 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-14T05:11:12.383 INFO:teuthology.orchestra.run.smithi194.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout:Recommended packages: 2024-09-14T05:11:12.384 INFO:teuthology.orchestra.run.smithi194.stdout: btrfs-tools 2024-09-14T05:11:12.480 INFO:teuthology.orchestra.run.smithi144.stdout:The following NEW packages will be installed: 2024-09-14T05:11:12.480 INFO:teuthology.orchestra.run.smithi144.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-14T05:11:12.480 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-14T05:11:12.481 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-14T05:11:12.481 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-14T05:11:12.481 INFO:teuthology.orchestra.run.smithi144.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-14T05:11:12.481 INFO:teuthology.orchestra.run.smithi144.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-14T05:11:12.484 INFO:teuthology.orchestra.run.smithi144.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-14T05:11:12.484 INFO:teuthology.orchestra.run.smithi144.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-14T05:11:12.484 INFO:teuthology.orchestra.run.smithi144.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-14T05:11:12.485 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-14T05:11:12.485 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-14T05:11:12.485 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-14T05:11:12.485 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-14T05:11:12.485 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-14T05:11:12.486 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-14T05:11:12.486 INFO:teuthology.orchestra.run.smithi144.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-14T05:11:12.486 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-14T05:11:12.486 INFO:teuthology.orchestra.run.smithi144.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-14T05:11:12.487 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-14T05:11:12.487 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-14T05:11:12.487 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-14T05:11:12.487 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-14T05:11:12.487 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-14T05:11:12.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-14T05:11:12.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-14T05:11:12.488 INFO:teuthology.orchestra.run.smithi144.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-14T05:11:12.491 INFO:teuthology.orchestra.run.smithi194.stdout:The following NEW packages will be installed: 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-14T05:11:12.492 INFO:teuthology.orchestra.run.smithi194.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-14T05:11:12.493 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-14T05:11:12.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-14T05:11:12.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-14T05:11:12.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-14T05:11:12.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-14T05:11:12.494 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-14T05:11:12.495 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-14T05:11:12.496 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-14T05:11:12.496 INFO:teuthology.orchestra.run.smithi194.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-14T05:11:12.541 INFO:teuthology.orchestra.run.smithi144.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-14T05:11:12.541 INFO:teuthology.orchestra.run.smithi144.stdout:Need to get 251 MB of archives. 2024-09-14T05:11:12.541 INFO:teuthology.orchestra.run.smithi144.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-14T05:11:12.542 INFO:teuthology.orchestra.run.smithi144.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-14T05:11:12.622 INFO:teuthology.orchestra.run.smithi144.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4775-g7f24e589-1jammy [3,631 kB] 2024-09-14T05:11:12.628 INFO:teuthology.orchestra.run.smithi194.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-14T05:11:12.628 INFO:teuthology.orchestra.run.smithi194.stdout:Need to get 251 MB of archives. 2024-09-14T05:11:12.628 INFO:teuthology.orchestra.run.smithi194.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-14T05:11:12.628 INFO:teuthology.orchestra.run.smithi194.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4775-g7f24e589-1jammy [3,631 kB] 2024-09-14T05:11:12.630 INFO:teuthology.orchestra.run.smithi144.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-14T05:11:12.632 INFO:teuthology.orchestra.run.smithi144.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-14T05:11:12.650 INFO:teuthology.orchestra.run.smithi144.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-14T05:11:12.673 INFO:teuthology.orchestra.run.smithi194.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-14T05:11:12.709 INFO:teuthology.orchestra.run.smithi144.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-14T05:11:12.713 INFO:teuthology.orchestra.run.smithi144.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-14T05:11:12.726 INFO:teuthology.orchestra.run.smithi144.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-14T05:11:12.730 INFO:teuthology.orchestra.run.smithi144.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-14T05:11:12.731 INFO:teuthology.orchestra.run.smithi144.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-14T05:11:12.731 INFO:teuthology.orchestra.run.smithi144.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-14T05:11:12.732 INFO:teuthology.orchestra.run.smithi144.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-14T05:11:12.741 INFO:teuthology.orchestra.run.smithi144.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-14T05:11:12.743 INFO:teuthology.orchestra.run.smithi144.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-14T05:11:12.745 INFO:teuthology.orchestra.run.smithi144.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-14T05:11:12.747 INFO:teuthology.orchestra.run.smithi144.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-14T05:11:12.747 INFO:teuthology.orchestra.run.smithi144.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-14T05:11:12.764 INFO:teuthology.orchestra.run.smithi144.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-14T05:11:12.766 INFO:teuthology.orchestra.run.smithi144.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-14T05:11:12.768 INFO:teuthology.orchestra.run.smithi144.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-14T05:11:12.769 INFO:teuthology.orchestra.run.smithi144.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-14T05:11:12.769 INFO:teuthology.orchestra.run.smithi144.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-14T05:11:12.770 INFO:teuthology.orchestra.run.smithi144.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-14T05:11:12.770 INFO:teuthology.orchestra.run.smithi144.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-14T05:11:12.771 INFO:teuthology.orchestra.run.smithi144.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-14T05:11:12.781 INFO:teuthology.orchestra.run.smithi144.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-14T05:11:12.782 INFO:teuthology.orchestra.run.smithi144.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-14T05:11:12.798 INFO:teuthology.orchestra.run.smithi144.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-14T05:11:12.799 INFO:teuthology.orchestra.run.smithi144.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-14T05:11:12.801 INFO:teuthology.orchestra.run.smithi144.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-14T05:11:12.802 INFO:teuthology.orchestra.run.smithi144.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-14T05:11:12.802 INFO:teuthology.orchestra.run.smithi144.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-14T05:11:12.803 INFO:teuthology.orchestra.run.smithi144.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-14T05:11:12.804 INFO:teuthology.orchestra.run.smithi144.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-14T05:11:12.804 INFO:teuthology.orchestra.run.smithi144.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-14T05:11:12.815 INFO:teuthology.orchestra.run.smithi144.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-14T05:11:12.817 INFO:teuthology.orchestra.run.smithi144.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-14T05:11:12.829 INFO:teuthology.orchestra.run.smithi194.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4775-g7f24e589-1jammy [3,330 kB] 2024-09-14T05:11:12.833 INFO:teuthology.orchestra.run.smithi144.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-14T05:11:12.833 INFO:teuthology.orchestra.run.smithi144.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-14T05:11:12.838 INFO:teuthology.orchestra.run.smithi144.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-14T05:11:12.839 INFO:teuthology.orchestra.run.smithi144.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-14T05:11:12.839 INFO:teuthology.orchestra.run.smithi144.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-14T05:11:12.840 INFO:teuthology.orchestra.run.smithi144.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-14T05:11:12.841 INFO:teuthology.orchestra.run.smithi144.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-14T05:11:12.841 INFO:teuthology.orchestra.run.smithi144.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4775-g7f24e589-1jammy [3,330 kB] 2024-09-14T05:11:12.843 INFO:teuthology.orchestra.run.smithi144.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-14T05:11:12.850 INFO:teuthology.orchestra.run.smithi144.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-14T05:11:12.851 INFO:teuthology.orchestra.run.smithi144.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-14T05:11:12.861 INFO:teuthology.orchestra.run.smithi144.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-14T05:11:12.863 INFO:teuthology.orchestra.run.smithi144.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-14T05:11:12.864 INFO:teuthology.orchestra.run.smithi144.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-14T05:11:12.865 INFO:teuthology.orchestra.run.smithi144.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-14T05:11:12.867 INFO:teuthology.orchestra.run.smithi144.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-14T05:11:12.917 INFO:teuthology.orchestra.run.smithi144.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4775-g7f24e589-1jammy [722 kB] 2024-09-14T05:11:12.932 INFO:teuthology.orchestra.run.smithi144.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-14T05:11:12.933 INFO:teuthology.orchestra.run.smithi144.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-14T05:11:12.934 INFO:teuthology.orchestra.run.smithi144.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-14T05:11:12.934 INFO:teuthology.orchestra.run.smithi144.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-14T05:11:12.935 INFO:teuthology.orchestra.run.smithi144.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-14T05:11:12.937 INFO:teuthology.orchestra.run.smithi144.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4775-g7f24e589-1jammy [356 kB] 2024-09-14T05:11:12.948 INFO:teuthology.orchestra.run.smithi194.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4775-g7f24e589-1jammy [722 kB] 2024-09-14T05:11:12.952 INFO:teuthology.orchestra.run.smithi144.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4775-g7f24e589-1jammy [32.8 kB] 2024-09-14T05:11:12.953 INFO:teuthology.orchestra.run.smithi144.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-14T05:11:12.953 INFO:teuthology.orchestra.run.smithi144.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4775-g7f24e589-1jammy [180 kB] 2024-09-14T05:11:12.955 INFO:teuthology.orchestra.run.smithi144.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-14T05:11:12.955 INFO:teuthology.orchestra.run.smithi144.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-14T05:11:12.956 INFO:teuthology.orchestra.run.smithi144.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4775-g7f24e589-1jammy [71.2 kB] 2024-09-14T05:11:12.958 INFO:teuthology.orchestra.run.smithi144.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4775-g7f24e589-1jammy [338 kB] 2024-09-14T05:11:12.964 INFO:teuthology.orchestra.run.smithi144.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4775-g7f24e589-1jammy [7,984 kB] 2024-09-14T05:11:12.971 INFO:teuthology.orchestra.run.smithi144.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-14T05:11:12.971 INFO:teuthology.orchestra.run.smithi144.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-14T05:11:12.972 INFO:teuthology.orchestra.run.smithi144.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-14T05:11:12.972 INFO:teuthology.orchestra.run.smithi144.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-14T05:11:12.973 INFO:teuthology.orchestra.run.smithi144.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-14T05:11:12.973 INFO:teuthology.orchestra.run.smithi144.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-14T05:11:12.976 INFO:teuthology.orchestra.run.smithi194.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4775-g7f24e589-1jammy [356 kB] 2024-09-14T05:11:12.985 INFO:teuthology.orchestra.run.smithi144.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-14T05:11:12.986 INFO:teuthology.orchestra.run.smithi144.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-14T05:11:12.989 INFO:teuthology.orchestra.run.smithi144.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-14T05:11:12.998 INFO:teuthology.orchestra.run.smithi194.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4775-g7f24e589-1jammy [32.8 kB] 2024-09-14T05:11:12.999 INFO:teuthology.orchestra.run.smithi194.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4775-g7f24e589-1jammy [180 kB] 2024-09-14T05:11:13.003 INFO:teuthology.orchestra.run.smithi194.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4775-g7f24e589-1jammy [71.2 kB] 2024-09-14T05:11:13.005 INFO:teuthology.orchestra.run.smithi144.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-14T05:11:13.005 INFO:teuthology.orchestra.run.smithi194.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4775-g7f24e589-1jammy [338 kB] 2024-09-14T05:11:13.007 INFO:teuthology.orchestra.run.smithi144.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-14T05:11:13.007 INFO:teuthology.orchestra.run.smithi144.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-14T05:11:13.011 INFO:teuthology.orchestra.run.smithi144.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-14T05:11:13.013 INFO:teuthology.orchestra.run.smithi144.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-14T05:11:13.014 INFO:teuthology.orchestra.run.smithi144.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-14T05:11:13.014 INFO:teuthology.orchestra.run.smithi144.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-14T05:11:13.021 INFO:teuthology.orchestra.run.smithi144.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-14T05:11:13.022 INFO:teuthology.orchestra.run.smithi144.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-14T05:11:13.025 INFO:teuthology.orchestra.run.smithi144.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-14T05:11:13.027 INFO:teuthology.orchestra.run.smithi194.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4775-g7f24e589-1jammy [7,984 kB] 2024-09-14T05:11:13.039 INFO:teuthology.orchestra.run.smithi144.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-14T05:11:13.039 INFO:teuthology.orchestra.run.smithi144.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-14T05:11:13.040 INFO:teuthology.orchestra.run.smithi144.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-14T05:11:13.041 INFO:teuthology.orchestra.run.smithi144.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-14T05:11:13.042 INFO:teuthology.orchestra.run.smithi144.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-14T05:11:13.048 INFO:teuthology.orchestra.run.smithi144.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-14T05:11:13.048 INFO:teuthology.orchestra.run.smithi144.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-14T05:11:13.056 INFO:teuthology.orchestra.run.smithi144.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-14T05:11:13.058 INFO:teuthology.orchestra.run.smithi144.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-14T05:11:13.059 INFO:teuthology.orchestra.run.smithi144.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-14T05:11:13.094 INFO:teuthology.orchestra.run.smithi194.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-14T05:11:13.106 INFO:teuthology.orchestra.run.smithi194.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-14T05:11:13.162 INFO:teuthology.orchestra.run.smithi144.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4775-g7f24e589-1jammy [112 kB] 2024-09-14T05:11:13.164 INFO:teuthology.orchestra.run.smithi144.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4775-g7f24e589-1jammy [473 kB] 2024-09-14T05:11:13.172 INFO:teuthology.orchestra.run.smithi144.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4775-g7f24e589-1jammy [28.3 MB] 2024-09-14T05:11:13.187 INFO:teuthology.orchestra.run.smithi194.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-14T05:11:13.416 INFO:teuthology.orchestra.run.smithi194.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-14T05:11:13.427 INFO:teuthology.orchestra.run.smithi194.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-14T05:11:13.457 INFO:teuthology.orchestra.run.smithi194.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-14T05:11:13.466 INFO:teuthology.orchestra.run.smithi194.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-14T05:11:13.468 INFO:teuthology.orchestra.run.smithi194.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-14T05:11:13.469 INFO:teuthology.orchestra.run.smithi194.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-14T05:11:13.470 INFO:teuthology.orchestra.run.smithi194.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-14T05:11:13.488 INFO:teuthology.orchestra.run.smithi194.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-14T05:11:13.492 INFO:teuthology.orchestra.run.smithi194.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-14T05:11:13.498 INFO:teuthology.orchestra.run.smithi194.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4775-g7f24e589-1jammy [112 kB] 2024-09-14T05:11:13.498 INFO:teuthology.orchestra.run.smithi194.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-14T05:11:13.528 INFO:teuthology.orchestra.run.smithi194.stdout:Get:25 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4775-g7f24e589-1jammy [473 kB] 2024-09-14T05:11:13.541 INFO:teuthology.orchestra.run.smithi194.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4775-g7f24e589-1jammy [28.3 MB] 2024-09-14T05:11:13.578 INFO:teuthology.orchestra.run.smithi194.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-14T05:11:13.579 INFO:teuthology.orchestra.run.smithi194.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-14T05:11:13.584 INFO:teuthology.orchestra.run.smithi194.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-14T05:11:13.589 INFO:teuthology.orchestra.run.smithi194.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-14T05:11:13.592 INFO:teuthology.orchestra.run.smithi194.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-14T05:11:13.594 INFO:teuthology.orchestra.run.smithi194.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-14T05:11:13.594 INFO:teuthology.orchestra.run.smithi194.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-14T05:11:13.596 INFO:teuthology.orchestra.run.smithi194.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-14T05:11:13.597 INFO:teuthology.orchestra.run.smithi194.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-14T05:11:13.662 INFO:teuthology.orchestra.run.smithi194.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-14T05:11:13.680 INFO:teuthology.orchestra.run.smithi194.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-14T05:11:13.680 INFO:teuthology.orchestra.run.smithi194.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-14T05:11:13.681 INFO:teuthology.orchestra.run.smithi194.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-14T05:11:13.682 INFO:teuthology.orchestra.run.smithi194.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-14T05:11:13.688 INFO:teuthology.orchestra.run.smithi194.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-14T05:11:13.744 INFO:teuthology.orchestra.run.smithi194.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-14T05:11:13.745 INFO:teuthology.orchestra.run.smithi194.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-14T05:11:13.745 INFO:teuthology.orchestra.run.smithi194.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-14T05:11:13.747 INFO:teuthology.orchestra.run.smithi194.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-14T05:11:13.825 INFO:teuthology.orchestra.run.smithi194.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-14T05:11:13.825 INFO:teuthology.orchestra.run.smithi194.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-14T05:11:13.827 INFO:teuthology.orchestra.run.smithi194.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-14T05:11:13.829 INFO:teuthology.orchestra.run.smithi194.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-14T05:11:13.830 INFO:teuthology.orchestra.run.smithi194.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-14T05:11:13.906 INFO:teuthology.orchestra.run.smithi194.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-14T05:11:13.907 INFO:teuthology.orchestra.run.smithi144.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4775-g7f24e589-1jammy [5,206 kB] 2024-09-14T05:11:13.907 INFO:teuthology.orchestra.run.smithi194.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-14T05:11:13.908 INFO:teuthology.orchestra.run.smithi194.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-14T05:11:13.909 INFO:teuthology.orchestra.run.smithi194.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-14T05:11:13.911 INFO:teuthology.orchestra.run.smithi194.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-14T05:11:13.986 INFO:teuthology.orchestra.run.smithi144.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4775-g7f24e589-1jammy [251 kB] 2024-09-14T05:11:13.988 INFO:teuthology.orchestra.run.smithi194.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-14T05:11:13.988 INFO:teuthology.orchestra.run.smithi144.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4775-g7f24e589-1jammy [124 kB] 2024-09-14T05:11:13.989 INFO:teuthology.orchestra.run.smithi194.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-14T05:11:13.990 INFO:teuthology.orchestra.run.smithi144.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4775-g7f24e589-1jammy [1,561 kB] 2024-09-14T05:11:13.992 INFO:teuthology.orchestra.run.smithi194.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-14T05:11:13.994 INFO:teuthology.orchestra.run.smithi194.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-14T05:11:13.999 INFO:teuthology.orchestra.run.smithi194.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-14T05:11:14.003 INFO:teuthology.orchestra.run.smithi144.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4775-g7f24e589-1jammy [6,311 kB] 2024-09-14T05:11:14.069 INFO:teuthology.orchestra.run.smithi194.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-14T05:11:14.071 INFO:teuthology.orchestra.run.smithi194.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-14T05:11:14.075 INFO:teuthology.orchestra.run.smithi194.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-14T05:11:14.129 INFO:teuthology.orchestra.run.smithi144.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4775-g7f24e589-1jammy [23.1 MB] 2024-09-14T05:11:14.239 INFO:teuthology.orchestra.run.smithi194.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-14T05:11:14.241 INFO:teuthology.orchestra.run.smithi194.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-14T05:11:14.243 INFO:teuthology.orchestra.run.smithi194.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-14T05:11:14.244 INFO:teuthology.orchestra.run.smithi194.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-14T05:11:14.246 INFO:teuthology.orchestra.run.smithi194.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-14T05:11:14.283 INFO:teuthology.orchestra.run.smithi194.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-14T05:11:14.286 INFO:teuthology.orchestra.run.smithi194.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-14T05:11:14.287 INFO:teuthology.orchestra.run.smithi194.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-14T05:11:14.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-14T05:11:14.344 INFO:teuthology.orchestra.run.smithi194.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-14T05:11:14.406 INFO:teuthology.orchestra.run.smithi194.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-14T05:11:14.407 INFO:teuthology.orchestra.run.smithi194.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-14T05:11:14.408 INFO:teuthology.orchestra.run.smithi194.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-14T05:11:14.409 INFO:teuthology.orchestra.run.smithi194.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-14T05:11:14.409 INFO:teuthology.orchestra.run.smithi194.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-14T05:11:14.410 INFO:teuthology.orchestra.run.smithi194.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-14T05:11:14.416 INFO:teuthology.orchestra.run.smithi194.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-14T05:11:14.488 INFO:teuthology.orchestra.run.smithi194.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-14T05:11:14.491 INFO:teuthology.orchestra.run.smithi194.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-14T05:11:14.492 INFO:teuthology.orchestra.run.smithi194.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-14T05:11:14.569 INFO:teuthology.orchestra.run.smithi194.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-14T05:11:14.575 INFO:teuthology.orchestra.run.smithi194.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-14T05:11:14.577 INFO:teuthology.orchestra.run.smithi194.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-14T05:11:14.579 INFO:teuthology.orchestra.run.smithi194.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-14T05:11:14.590 INFO:teuthology.orchestra.run.smithi194.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-14T05:11:14.592 INFO:teuthology.orchestra.run.smithi194.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-14T05:11:14.599 INFO:teuthology.orchestra.run.smithi194.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-14T05:11:14.650 INFO:teuthology.orchestra.run.smithi194.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-14T05:11:14.652 INFO:teuthology.orchestra.run.smithi194.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-14T05:11:14.653 INFO:teuthology.orchestra.run.smithi194.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-14T05:11:14.682 INFO:teuthology.orchestra.run.smithi144.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4775-g7f24e589-1jammy [14.1 kB] 2024-09-14T05:11:14.682 INFO:teuthology.orchestra.run.smithi144.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4775-g7f24e589-1jammy [880 kB] 2024-09-14T05:11:14.695 INFO:teuthology.orchestra.run.smithi144.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4775-g7f24e589-1jammy [2,452 kB] 2024-09-14T05:11:14.729 INFO:teuthology.orchestra.run.smithi144.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4775-g7f24e589-1jammy [812 kB] 2024-09-14T05:11:14.732 INFO:teuthology.orchestra.run.smithi194.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-14T05:11:14.734 INFO:teuthology.orchestra.run.smithi194.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-14T05:11:14.739 INFO:teuthology.orchestra.run.smithi144.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4775-g7f24e589-1jammy [160 kB] 2024-09-14T05:11:14.741 INFO:teuthology.orchestra.run.smithi144.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4775-g7f24e589-1jammy [61.4 MB] 2024-09-14T05:11:14.751 INFO:teuthology.orchestra.run.smithi194.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-14T05:11:14.751 INFO:teuthology.orchestra.run.smithi194.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-14T05:11:14.752 INFO:teuthology.orchestra.run.smithi194.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-14T05:11:14.757 INFO:teuthology.orchestra.run.smithi194.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-14T05:11:14.759 INFO:teuthology.orchestra.run.smithi194.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-14T05:11:15.697 INFO:teuthology.orchestra.run.smithi194.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4775-g7f24e589-1jammy [5,206 kB] 2024-09-14T05:11:16.036 INFO:teuthology.orchestra.run.smithi194.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4775-g7f24e589-1jammy [251 kB] 2024-09-14T05:11:16.041 INFO:teuthology.orchestra.run.smithi194.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4775-g7f24e589-1jammy [124 kB] 2024-09-14T05:11:16.044 INFO:teuthology.orchestra.run.smithi194.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4775-g7f24e589-1jammy [1,561 kB] 2024-09-14T05:11:16.086 INFO:teuthology.orchestra.run.smithi194.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4775-g7f24e589-1jammy [6,311 kB] 2024-09-14T05:11:16.221 INFO:teuthology.orchestra.run.smithi144.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4775-g7f24e589-1jammy [8,625 kB] 2024-09-14T05:11:16.266 INFO:teuthology.orchestra.run.smithi194.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4775-g7f24e589-1jammy [23.1 MB] 2024-09-14T05:11:16.351 INFO:teuthology.orchestra.run.smithi144.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4775-g7f24e589-1jammy [14.2 kB] 2024-09-14T05:11:16.352 INFO:teuthology.orchestra.run.smithi144.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4775-g7f24e589-1jammy [55.8 MB] 2024-09-14T05:11:17.767 INFO:teuthology.orchestra.run.smithi144.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4775-g7f24e589-1jammy [130 kB] 2024-09-14T05:11:17.769 INFO:teuthology.orchestra.run.smithi144.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4775-g7f24e589-1jammy [39.0 kB] 2024-09-14T05:11:17.770 INFO:teuthology.orchestra.run.smithi144.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4775-g7f24e589-1jammy [15.9 MB] 2024-09-14T05:11:17.840 INFO:teuthology.orchestra.run.smithi194.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4775-g7f24e589-1jammy [14.1 kB] 2024-09-14T05:11:17.840 INFO:teuthology.orchestra.run.smithi194.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4775-g7f24e589-1jammy [880 kB] 2024-09-14T05:11:17.857 INFO:teuthology.orchestra.run.smithi194.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4775-g7f24e589-1jammy [2,452 kB] 2024-09-14T05:11:17.980 INFO:teuthology.orchestra.run.smithi194.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4775-g7f24e589-1jammy [812 kB] 2024-09-14T05:11:18.037 INFO:teuthology.orchestra.run.smithi194.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4775-g7f24e589-1jammy [160 kB] 2024-09-14T05:11:18.040 INFO:teuthology.orchestra.run.smithi194.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4775-g7f24e589-1jammy [61.4 MB] 2024-09-14T05:11:18.113 INFO:teuthology.orchestra.run.smithi144.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4775-g7f24e589-1jammy [91.3 kB] 2024-09-14T05:11:18.673 INFO:teuthology.orchestra.run.smithi144.stdout:Fetched 251 MB in 6s (44.6 MB/s) 2024-09-14T05:11:18.831 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-14T05:11:18.926 INFO:teuthology.orchestra.run.smithi144.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-14T05:11:18.931 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-14T05:11:19.063 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-14T05:11:19.558 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-14T05:11:19.584 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-14T05:11:19.625 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-14T05:11:19.920 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-14T05:11:19.946 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-14T05:11:19.950 INFO:teuthology.orchestra.run.smithi194.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4775-g7f24e589-1jammy [8,625 kB] 2024-09-14T05:11:19.978 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-14T05:11:20.098 INFO:teuthology.orchestra.run.smithi194.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4775-g7f24e589-1jammy [14.2 kB] 2024-09-14T05:11:20.099 INFO:teuthology.orchestra.run.smithi194.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4775-g7f24e589-1jammy [55.8 MB] 2024-09-14T05:11:20.415 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-14T05:11:20.442 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:20.489 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:20.868 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-14T05:11:20.894 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:20.934 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:21.273 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-14T05:11:21.298 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:21.338 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:21.666 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-14T05:11:21.692 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-14T05:11:21.732 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-14T05:11:21.849 INFO:teuthology.orchestra.run.smithi194.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4775-g7f24e589-1jammy [130 kB] 2024-09-14T05:11:21.851 INFO:teuthology.orchestra.run.smithi194.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4775-g7f24e589-1jammy [39.0 kB] 2024-09-14T05:11:21.853 INFO:teuthology.orchestra.run.smithi194.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4775-g7f24e589-1jammy [15.9 MB] 2024-09-14T05:11:22.010 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librados2. 2024-09-14T05:11:22.036 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../007-librados2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:22.077 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librados2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:22.274 INFO:teuthology.orchestra.run.smithi194.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4775-g7f24e589-1jammy [91.3 kB] 2024-09-14T05:11:22.580 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libnbd0. 2024-09-14T05:11:22.606 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-14T05:11:22.646 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-14T05:11:22.862 INFO:teuthology.orchestra.run.smithi194.stdout:Fetched 251 MB in 10s (25.7 MB/s) 2024-09-14T05:11:22.966 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librbd1. 2024-09-14T05:11:22.992 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../009-librbd1_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:23.002 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-14T05:11:23.032 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librbd1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:23.102 INFO:teuthology.orchestra.run.smithi194.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-14T05:11:23.108 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-14T05:11:23.176 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-14T05:11:23.452 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs2. 2024-09-14T05:11:23.467 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:23.502 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:23.505 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-14T05:11:23.531 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-14T05:11:23.571 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-14T05:11:23.805 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rados. 2024-09-14T05:11:23.820 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:23.863 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rados (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:23.891 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-14T05:11:23.917 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-14T05:11:23.957 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-14T05:11:24.391 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-14T05:11:24.402 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-14T05:11:24.418 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:24.429 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:24.457 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-argparse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:24.477 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:24.793 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cephfs. 2024-09-14T05:11:24.809 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:24.851 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cephfs (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:24.855 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-14T05:11:24.881 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:24.922 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:25.106 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-14T05:11:25.131 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:25.163 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:25.474 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-14T05:11:25.500 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-14T05:11:25.533 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-14T05:11:25.540 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-14T05:11:25.559 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-14T05:11:25.600 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:11:25.843 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-prettytable. 2024-09-14T05:11:25.869 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-14T05:11:25.910 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-14T05:11:25.961 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-14T05:11:25.987 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-14T05:11:26.028 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-14T05:11:26.213 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rbd. 2024-09-14T05:11:26.239 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:26.279 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rbd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:26.305 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librados2. 2024-09-14T05:11:26.332 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../007-librados2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:26.389 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librados2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:26.582 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-14T05:11:26.609 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-14T05:11:26.641 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-14T05:11:26.866 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libnbd0. 2024-09-14T05:11:26.892 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-14T05:11:26.927 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-14T05:11:26.933 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-14T05:11:26.953 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-14T05:11:26.993 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-14T05:11:27.229 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librbd1. 2024-09-14T05:11:27.256 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../009-librbd1_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:27.279 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-14T05:11:27.294 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-14T05:11:27.296 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librbd1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:27.338 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-14T05:11:27.716 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua5.1. 2024-09-14T05:11:27.732 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libcephfs2. 2024-09-14T05:11:27.742 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-14T05:11:27.758 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:27.791 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-14T05:11:27.798 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libcephfs2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:28.076 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rados. 2024-09-14T05:11:28.091 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:28.126 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rados (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:28.136 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-any. 2024-09-14T05:11:28.151 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-14T05:11:28.194 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-14T05:11:28.403 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-14T05:11:28.429 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:28.439 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package zip. 2024-09-14T05:11:28.454 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-14T05:11:28.470 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-ceph-argparse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:28.497 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking zip (3.0-12build2) ... 2024-09-14T05:11:28.722 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cephfs. 2024-09-14T05:11:28.749 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:28.789 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cephfs (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:28.809 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package unzip. 2024-09-14T05:11:28.835 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-14T05:11:28.883 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-14T05:11:29.100 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-14T05:11:29.126 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:29.166 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:29.329 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package luarocks. 2024-09-14T05:11:29.356 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-14T05:11:29.396 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-14T05:11:29.486 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-14T05:11:29.501 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-14T05:11:29.544 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-14T05:11:29.795 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package librgw2. 2024-09-14T05:11:29.822 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../026-librgw2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:29.864 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking librgw2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:29.865 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-prettytable. 2024-09-14T05:11:29.890 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-14T05:11:29.922 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-14T05:11:30.174 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rbd. 2024-09-14T05:11:30.190 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:30.233 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rbd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:30.481 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rgw. 2024-09-14T05:11:30.496 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:30.531 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:30.611 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-14T05:11:30.637 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-14T05:11:30.677 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-14T05:11:30.809 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-14T05:11:30.835 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-14T05:11:30.875 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-14T05:11:30.922 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-14T05:11:30.948 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-14T05:11:30.988 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-14T05:11:31.204 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libradosstriper1. 2024-09-14T05:11:31.230 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:31.270 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libradosstriper1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:31.282 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-14T05:11:31.309 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-14T05:11:31.349 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-14T05:11:31.669 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package lua5.1. 2024-09-14T05:11:31.682 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-common. 2024-09-14T05:11:31.695 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-14T05:11:31.708 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:31.735 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-14T05:11:31.748 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:32.047 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package lua-any. 2024-09-14T05:11:32.073 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-14T05:11:32.105 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-14T05:11:32.332 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package zip. 2024-09-14T05:11:32.348 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-14T05:11:32.390 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking zip (3.0-12build2) ... 2024-09-14T05:11:32.720 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package unzip. 2024-09-14T05:11:32.745 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-14T05:11:32.785 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-14T05:11:33.189 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package luarocks. 2024-09-14T05:11:33.215 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-14T05:11:33.255 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-14T05:11:33.262 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-base. 2024-09-14T05:11:33.288 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:33.347 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-base (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:33.654 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package librgw2. 2024-09-14T05:11:33.680 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../026-librgw2_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:33.729 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking librgw2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:33.941 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-14T05:11:33.968 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-14T05:11:34.008 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-14T05:11:34.307 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rgw. 2024-09-14T05:11:34.322 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:34.356 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:34.546 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cheroot. 2024-09-14T05:11:34.573 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-14T05:11:34.626 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-14T05:11:34.627 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-14T05:11:34.642 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-14T05:11:34.675 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-14T05:11:34.964 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-14T05:11:34.987 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libradosstriper1. 2024-09-14T05:11:34.990 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-14T05:11:35.013 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:35.030 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-14T05:11:35.053 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libradosstriper1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:35.317 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-14T05:11:35.343 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-14T05:11:35.383 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-14T05:11:35.473 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-common. 2024-09-14T05:11:35.489 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:35.531 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:35.728 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-14T05:11:35.754 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-14T05:11:35.803 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-14T05:11:36.147 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tempora. 2024-09-14T05:11:36.174 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-14T05:11:36.214 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-14T05:11:36.517 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-portend. 2024-09-14T05:11:36.532 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-14T05:11:36.575 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-14T05:11:36.937 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-base. 2024-09-14T05:11:36.953 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-14T05:11:36.964 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:36.980 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-14T05:11:37.004 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-base (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:37.028 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-14T05:11:37.398 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-14T05:11:37.424 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-14T05:11:37.472 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-14T05:11:37.599 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-14T05:11:37.625 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-14T05:11:37.665 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-14T05:11:37.801 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-natsort. 2024-09-14T05:11:37.828 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-14T05:11:37.859 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-14T05:11:37.968 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cheroot. 2024-09-14T05:11:37.994 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-14T05:11:38.034 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-14T05:11:38.138 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-logutils. 2024-09-14T05:11:38.164 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-14T05:11:38.204 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-14T05:11:38.354 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-14T05:11:38.380 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-14T05:11:38.421 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-14T05:11:38.507 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-14T05:11:38.534 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-14T05:11:38.574 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-14T05:11:38.715 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-14T05:11:38.742 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-14T05:11:38.782 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-14T05:11:38.860 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-mako. 2024-09-14T05:11:38.886 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-14T05:11:38.926 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-14T05:11:39.084 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-14T05:11:39.111 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-14T05:11:39.151 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-14T05:11:39.255 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-14T05:11:39.271 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-14T05:11:39.313 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-14T05:11:39.454 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-tempora. 2024-09-14T05:11:39.480 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-14T05:11:39.521 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-14T05:11:39.616 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-14T05:11:39.642 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-14T05:11:39.690 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-14T05:11:39.823 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-portend. 2024-09-14T05:11:39.849 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-14T05:11:39.881 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-14T05:11:39.952 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webob. 2024-09-14T05:11:39.968 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-14T05:11:40.002 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-14T05:11:40.167 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-14T05:11:40.193 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-14T05:11:40.233 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-14T05:11:40.314 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-waitress. 2024-09-14T05:11:40.340 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-14T05:11:40.383 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-14T05:11:40.536 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-14T05:11:40.563 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-14T05:11:40.603 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-14T05:11:40.758 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-tempita. 2024-09-14T05:11:40.785 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-14T05:11:40.825 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-14T05:11:41.128 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-paste. 2024-09-14T05:11:41.148 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-natsort. 2024-09-14T05:11:41.154 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-14T05:11:41.175 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-14T05:11:41.194 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-14T05:11:41.214 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-14T05:11:41.493 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-logutils. 2024-09-14T05:11:41.514 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-14T05:11:41.519 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-14T05:11:41.541 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-14T05:11:41.559 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-14T05:11:41.581 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-14T05:11:41.887 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-14T05:11:41.900 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-14T05:11:41.914 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-14T05:11:41.927 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-14T05:11:41.953 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-14T05:11:41.967 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-14T05:11:42.254 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-webtest. 2024-09-14T05:11:42.256 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-mako. 2024-09-14T05:11:42.280 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-14T05:11:42.283 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-14T05:11:42.311 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-14T05:11:42.323 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-14T05:11:42.556 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pecan. 2024-09-14T05:11:42.579 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-14T05:11:42.623 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-14T05:11:42.668 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-14T05:11:42.694 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-14T05:11:42.734 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-14T05:11:42.986 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-certifi. 2024-09-14T05:11:43.003 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-14T05:11:43.013 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-14T05:11:43.019 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-14T05:11:43.052 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-14T05:11:43.053 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-14T05:11:43.356 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-webob. 2024-09-14T05:11:43.381 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-idna. 2024-09-14T05:11:43.382 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-14T05:11:43.407 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-14T05:11:43.422 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-14T05:11:43.447 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-14T05:11:43.767 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-waitress. 2024-09-14T05:11:43.767 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-urllib3. 2024-09-14T05:11:43.783 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-14T05:11:43.793 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-14T05:11:43.828 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-14T05:11:43.833 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-14T05:11:44.136 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-tempita. 2024-09-14T05:11:44.163 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-14T05:11:44.187 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-requests. 2024-09-14T05:11:44.203 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-14T05:11:44.213 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-14T05:11:44.253 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-14T05:11:44.514 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-paste. 2024-09-14T05:11:44.540 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-14T05:11:44.540 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-14T05:11:44.566 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-14T05:11:44.580 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-14T05:11:44.598 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-14T05:11:45.043 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-14T05:11:45.069 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:45.117 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:45.523 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-14T05:11:45.550 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:45.589 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:45.676 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-14T05:11:45.703 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-14T05:11:45.742 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-14T05:11:46.020 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-14T05:11:46.046 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-14T05:11:46.086 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-14T05:11:46.373 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-webtest. 2024-09-14T05:11:46.399 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-14T05:11:46.439 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-14T05:11:46.643 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-14T05:11:46.670 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-14T05:11:46.750 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pecan. 2024-09-14T05:11:46.777 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-14T05:11:46.817 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-14T05:11:46.817 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-14T05:11:47.146 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-openssl. 2024-09-14T05:11:47.171 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-certifi. 2024-09-14T05:11:47.172 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-14T05:11:47.198 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-14T05:11:47.212 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-14T05:11:47.246 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-14T05:11:47.549 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr. 2024-09-14T05:11:47.574 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-idna. 2024-09-14T05:11:47.575 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:47.601 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-14T05:11:47.615 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:47.640 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-14T05:11:47.943 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-urllib3. 2024-09-14T05:11:47.959 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-14T05:11:47.993 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-14T05:11:48.069 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mon. 2024-09-14T05:11:48.095 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:48.143 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mon (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:48.296 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-requests. 2024-09-14T05:11:48.323 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-14T05:11:48.362 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-14T05:11:48.649 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-14T05:11:48.676 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-14T05:11:48.715 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-14T05:11:48.766 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-osd. 2024-09-14T05:11:48.792 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:48.832 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-osd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:48.960 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-14T05:11:48.977 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:49.018 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:49.423 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-14T05:11:49.450 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:49.489 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:49.784 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-14T05:11:49.811 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-14T05:11:49.842 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-14T05:11:50.103 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-openssl. 2024-09-14T05:11:50.117 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph. 2024-09-14T05:11:50.118 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-14T05:11:50.143 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../067-ceph_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:50.153 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-14T05:11:50.193 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:50.489 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr. 2024-09-14T05:11:50.516 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:50.556 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:50.588 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-fuse. 2024-09-14T05:11:50.614 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:50.662 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:50.975 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mon. 2024-09-14T05:11:51.002 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:51.024 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mds. 2024-09-14T05:11:51.042 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mon (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:51.051 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:51.091 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mds (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:51.485 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package cephadm. 2024-09-14T05:11:51.512 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../070-cephadm_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:51.560 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:51.612 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-osd. 2024-09-14T05:11:51.638 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:51.670 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-osd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:51.946 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-14T05:11:51.973 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-14T05:11:52.029 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-14T05:11:52.399 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-14T05:11:52.426 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-14T05:11:52.466 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-14T05:11:52.715 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph. 2024-09-14T05:11:52.730 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../067-ceph_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:52.773 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:53.092 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-fuse. 2024-09-14T05:11:53.121 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:53.158 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:53.454 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-babel. 2024-09-14T05:11:53.481 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-14T05:11:53.520 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-14T05:11:53.528 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mds. 2024-09-14T05:11:53.554 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:53.594 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mds (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:53.849 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-jinja2. 2024-09-14T05:11:53.876 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-14T05:11:53.915 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-14T05:11:54.010 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package cephadm. 2024-09-14T05:11:54.037 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../070-cephadm_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:11:54.080 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:54.252 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-14T05:11:54.278 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:54.318 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:54.366 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-14T05:11:54.382 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-14T05:11:54.416 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-14T05:11:54.646 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-14T05:11:54.673 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-14T05:11:54.713 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-14T05:11:54.735 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-14T05:11:54.753 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-14T05:11:54.793 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-14T05:11:55.041 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-routes. 2024-09-14T05:11:55.068 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-14T05:11:55.107 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-14T05:11:55.453 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-14T05:11:55.479 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:55.519 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:55.857 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-babel. 2024-09-14T05:11:55.884 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-14T05:11:55.923 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-14T05:11:56.226 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-jinja2. 2024-09-14T05:11:56.253 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-14T05:11:56.341 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-14T05:11:56.787 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-14T05:11:56.814 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:56.853 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:57.216 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-14T05:11:57.243 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-14T05:11:57.282 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-14T05:11:57.559 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-routes. 2024-09-14T05:11:57.575 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-14T05:11:57.609 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-14T05:11:57.938 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-14T05:11:57.964 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:11:58.004 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:11:58.257 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-14T05:11:58.275 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-14T05:11:58.315 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-14T05:11:58.853 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-joblib. 2024-09-14T05:11:58.880 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-14T05:11:58.919 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-14T05:11:59.273 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-14T05:11:59.300 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-14T05:11:59.339 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-14T05:11:59.650 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-sklearn. 2024-09-14T05:11:59.678 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-14T05:11:59.811 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-14T05:12:00.517 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-14T05:12:00.546 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-14T05:12:00.591 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-14T05:12:00.795 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-14T05:12:00.823 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:00.862 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:01.028 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-joblib. 2024-09-14T05:12:01.055 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-14T05:12:01.086 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-14T05:12:01.431 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-14T05:12:01.447 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-14T05:12:01.489 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-14T05:12:01.767 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-sklearn. 2024-09-14T05:12:01.794 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-14T05:12:01.824 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-cachetools. 2024-09-14T05:12:01.851 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-14T05:12:01.890 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-14T05:12:01.919 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-14T05:12:02.159 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-14T05:12:02.175 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-14T05:12:02.218 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-14T05:12:02.530 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-14T05:12:02.557 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-14T05:12:02.596 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-14T05:12:02.603 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-14T05:12:02.631 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:02.899 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:02.916 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-rsa. 2024-09-14T05:12:02.945 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-14T05:12:02.982 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-14T05:12:03.278 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-google-auth. 2024-09-14T05:12:03.306 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-14T05:12:03.344 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-14T05:12:03.641 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-cachetools. 2024-09-14T05:12:03.665 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-14T05:12:03.668 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-14T05:12:03.692 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-14T05:12:03.707 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-14T05:12:03.731 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-14T05:12:04.010 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-14T05:12:04.034 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-websocket. 2024-09-14T05:12:04.037 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-14T05:12:04.062 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-14T05:12:04.077 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-14T05:12:04.101 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-14T05:12:04.387 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-14T05:12:04.405 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-14T05:12:04.414 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-14T05:12:04.432 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-14T05:12:04.445 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-14T05:12:04.471 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-14T05:12:04.791 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-rsa. 2024-09-14T05:12:04.819 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-14T05:12:04.857 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-14T05:12:05.109 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-14T05:12:05.138 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:05.169 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-google-auth. 2024-09-14T05:12:05.176 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:05.197 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-14T05:12:05.236 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-14T05:12:05.487 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-14T05:12:05.515 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-14T05:12:05.530 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-14T05:12:05.558 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-14T05:12:05.578 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-14T05:12:05.588 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-14T05:12:05.884 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-websocket. 2024-09-14T05:12:05.911 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-14T05:12:05.915 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-14T05:12:05.943 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-14T05:12:05.950 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-14T05:12:05.982 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-14T05:12:06.209 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package jq. 2024-09-14T05:12:06.225 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-14T05:12:06.267 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-14T05:12:06.270 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-14T05:12:06.297 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-14T05:12:06.453 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-14T05:12:06.671 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package socat. 2024-09-14T05:12:06.698 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-14T05:12:06.737 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-14T05:12:07.023 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package xmlstarlet. 2024-09-14T05:12:07.039 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-14T05:12:07.082 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-14T05:12:07.150 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-14T05:12:07.166 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:07.208 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:07.411 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-test. 2024-09-14T05:12:07.427 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:07.469 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-test (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:07.511 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-14T05:12:07.538 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-14T05:12:07.577 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-14T05:12:07.913 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-14T05:12:07.941 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-14T05:12:07.979 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-14T05:12:08.232 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package jq. 2024-09-14T05:12:08.248 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-14T05:12:08.290 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-14T05:12:08.535 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package socat. 2024-09-14T05:12:08.551 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-14T05:12:08.593 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-14T05:12:08.854 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package xmlstarlet. 2024-09-14T05:12:08.870 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-14T05:12:08.904 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-14T05:12:09.242 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-test. 2024-09-14T05:12:09.269 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:09.308 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-test (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:10.040 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package ceph-volume. 2024-09-14T05:12:10.069 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:10.106 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking ceph-volume (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:10.377 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-14T05:12:10.390 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:10.418 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking libcephfs-dev (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:10.747 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-14T05:12:10.775 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-14T05:12:10.813 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-14T05:12:11.159 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-14T05:12:11.187 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-14T05:12:11.226 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-14T05:12:11.547 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package nvme-cli. 2024-09-14T05:12:11.574 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-14T05:12:11.604 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-14T05:12:11.929 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package ceph-volume. 2024-09-14T05:12:11.957 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4775-g7f24e589-1jammy_all.deb ... 2024-09-14T05:12:11.968 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package pkg-config. 2024-09-14T05:12:11.995 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking ceph-volume (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:11.996 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-14T05:12:12.026 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-14T05:12:12.274 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-14T05:12:12.279 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-14T05:12:12.295 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-14T05:12:12.301 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:12.337 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-14T05:12:12.340 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking libcephfs-dev (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:12.677 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-14T05:12:12.704 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-14T05:12:12.744 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-14T05:12:12.791 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-14T05:12:12.819 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-14T05:12:12.857 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-14T05:12:13.064 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-14T05:12:13.092 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-14T05:12:13.130 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-14T05:12:13.160 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pastescript. 2024-09-14T05:12:13.176 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-14T05:12:13.218 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-14T05:12:13.484 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package nvme-cli. 2024-09-14T05:12:13.513 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-14T05:12:13.522 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pluggy. 2024-09-14T05:12:13.550 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-14T05:12:13.550 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-14T05:12:13.588 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-14T05:12:13.866 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-psutil. 2024-09-14T05:12:13.895 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-14T05:12:13.924 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-14T05:12:13.955 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package pkg-config. 2024-09-14T05:12:13.984 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-14T05:12:14.013 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-14T05:12:14.258 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-14T05:12:14.269 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-py. 2024-09-14T05:12:14.275 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-14T05:12:14.297 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-14T05:12:14.316 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-14T05:12:14.344 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-14T05:12:14.648 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pygments. 2024-09-14T05:12:14.664 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-14T05:12:14.706 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-14T05:12:14.711 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-14T05:12:14.739 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-14T05:12:14.777 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-14T05:12:15.080 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pastescript. 2024-09-14T05:12:15.093 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-14T05:12:15.108 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-14T05:12:15.110 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-14T05:12:15.143 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-14T05:12:15.146 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-14T05:12:15.462 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-toml. 2024-09-14T05:12:15.467 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pluggy. 2024-09-14T05:12:15.491 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-14T05:12:15.495 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-14T05:12:15.529 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-14T05:12:15.549 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-14T05:12:15.861 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-psutil. 2024-09-14T05:12:15.889 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-14T05:12:15.927 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-14T05:12:16.079 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-pytest. 2024-09-14T05:12:16.107 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-14T05:12:16.157 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-14T05:12:16.263 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-py. 2024-09-14T05:12:16.292 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-14T05:12:16.338 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-14T05:12:16.519 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package python3-simplejson. 2024-09-14T05:12:16.547 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-14T05:12:16.585 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-14T05:12:16.633 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pygments. 2024-09-14T05:12:16.661 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-14T05:12:16.691 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-14T05:12:16.855 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-14T05:12:16.883 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-14T05:12:16.921 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-14T05:12:17.179 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-14T05:12:17.207 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-14T05:12:17.245 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-14T05:12:17.548 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-toml. 2024-09-14T05:12:17.576 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-14T05:12:17.614 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-14T05:12:17.701 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package radosgw. 2024-09-14T05:12:17.728 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../117-radosgw_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:17.766 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking radosgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:17.917 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-pytest. 2024-09-14T05:12:17.946 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-14T05:12:17.992 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-14T05:12:18.329 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package python3-simplejson. 2024-09-14T05:12:18.357 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-14T05:12:18.395 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-14T05:12:18.689 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-14T05:12:18.705 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-14T05:12:18.745 INFO:teuthology.orchestra.run.smithi144.stdout:Selecting previously unselected package rbd-fuse. 2024-09-14T05:12:18.747 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-14T05:12:18.761 INFO:teuthology.orchestra.run.smithi144.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:18.803 INFO:teuthology.orchestra.run.smithi144.stdout:Unpacking rbd-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:19.096 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-14T05:12:19.350 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-14T05:12:19.443 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package radosgw. 2024-09-14T05:12:19.451 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-14T05:12:19.471 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../117-radosgw_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:19.509 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking radosgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:19.686 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-14T05:12:20.054 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-14T05:12:20.420 INFO:teuthology.orchestra.run.smithi194.stdout:Selecting previously unselected package rbd-fuse. 2024-09-14T05:12:20.437 INFO:teuthology.orchestra.run.smithi194.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4775-g7f24e589-1jammy_amd64.deb ... 2024-09-14T05:12:20.478 INFO:teuthology.orchestra.run.smithi194.stdout:Unpacking rbd-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:20.488 INFO:teuthology.orchestra.run.smithi144.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-14T05:12:20.553 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:20.774 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-14T05:12:21.033 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-14T05:12:21.069 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user cephadm....done 2024-09-14T05:12:21.168 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-14T05:12:21.426 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-14T05:12:21.432 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-14T05:12:21.694 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-14T05:12:21.775 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-14T05:12:21.953 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-14T05:12:22.063 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-14T05:12:22.219 INFO:teuthology.orchestra.run.smithi194.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-14T05:12:22.311 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:22.314 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-14T05:12:22.574 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-14T05:12:22.699 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-14T05:12:22.993 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-14T05:12:23.137 INFO:teuthology.orchestra.run.smithi194.stdout:Adding system user cephadm....done 2024-09-14T05:12:23.284 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-14T05:12:23.353 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-14T05:12:23.552 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-14T05:12:23.604 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-14T05:12:23.812 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-14T05:12:23.873 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-14T05:12:23.938 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-14T05:12:24.116 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-14T05:12:24.189 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-14T05:12:24.350 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-14T05:12:24.440 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-14T05:12:24.574 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-14T05:12:24.601 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-14T05:12:24.869 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-argparse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:24.876 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-14T05:12:25.129 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-14T05:12:25.253 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-14T05:12:25.254 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-14T05:12:25.389 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-14T05:12:25.504 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-14T05:12:25.515 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-14T05:12:25.633 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-14T05:12:25.772 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-14T05:12:25.867 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-14T05:12:26.065 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-14T05:12:26.219 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-14T05:12:26.248 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-14T05:12:26.312 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-14T05:12:26.316 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-14T05:12:26.413 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-14T05:12:26.651 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-14T05:12:26.781 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-14T05:12:26.901 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-ceph-argparse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:26.907 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-14T05:12:27.158 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-14T05:12:27.177 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-14T05:12:27.320 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-14T05:12:27.409 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-14T05:12:27.446 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-14T05:12:27.572 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-14T05:12:27.681 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-14T05:12:27.718 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up zip (3.0-12build2) ... 2024-09-14T05:12:27.845 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-14T05:12:27.932 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-14T05:12:28.422 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-14T05:12:28.455 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-14T05:12:28.519 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-14T05:12:28.530 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-14T05:12:28.645 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-14T05:12:28.798 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-14T05:12:28.924 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-14T05:12:29.012 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-14T05:12:29.033 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-14T05:12:29.138 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-14T05:12:29.318 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-14T05:12:29.406 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-14T05:12:29.665 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-14T05:12:29.702 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-14T05:12:29.953 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-14T05:12:29.975 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up zip (3.0-12build2) ... 2024-09-14T05:12:30.101 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-14T05:12:30.321 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-14T05:12:30.590 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-14T05:12:30.778 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-14T05:12:30.949 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-14T05:12:31.062 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-14T05:12:31.179 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-14T05:12:31.192 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-14T05:12:31.293 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:31.313 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-14T05:12:31.590 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-14T05:12:31.594 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-14T05:12:31.945 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-14T05:12:31.974 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-14T05:12:32.225 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-14T05:12:32.610 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-14T05:12:32.911 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-14T05:12:33.127 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-14T05:12:33.246 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-14T05:12:33.386 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:33.496 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-14T05:12:33.497 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-14T05:12:33.623 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:33.741 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-14T05:12:33.844 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-14T05:12:33.943 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-14T05:12:33.958 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-14T05:12:34.269 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-14T05:12:34.276 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-14T05:12:34.545 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-14T05:12:34.779 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:34.880 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-14T05:12:35.181 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-14T05:12:35.433 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-14T05:12:35.439 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-14T05:12:35.675 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-14T05:12:35.691 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:35.825 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-14T05:12:35.926 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-14T05:12:36.084 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-14T05:12:36.186 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-14T05:12:36.210 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-14T05:12:36.328 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-14T05:12:36.453 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-14T05:12:36.579 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-14T05:12:36.613 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-14T05:12:36.856 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-14T05:12:36.863 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-14T05:12:36.982 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-14T05:12:37.114 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:37.241 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-14T05:12:37.283 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-14T05:12:37.550 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-14T05:12:37.559 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-14T05:12:37.818 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-14T05:12:37.827 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-14T05:12:37.953 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-14T05:12:38.069 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-14T05:12:38.078 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-14T05:12:38.168 INFO:teuthology.orchestra.run.smithi144.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-14T05:12:38.303 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-14T05:12:38.413 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-14T05:12:38.537 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-14T05:12:38.539 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-14T05:12:38.805 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-14T05:12:38.931 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-14T05:12:38.941 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-14T05:12:39.184 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-14T05:12:39.216 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-14T05:12:39.310 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-14T05:12:39.342 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-14T05:12:39.578 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:39.634 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-14T05:12:39.704 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-14T05:12:39.927 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-14T05:12:40.013 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-14T05:12:40.173 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-14T05:12:40.296 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-14T05:12:40.306 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-14T05:12:40.422 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-14T05:12:40.511 INFO:teuthology.orchestra.run.smithi194.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-14T05:12:40.582 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-14T05:12:40.708 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-14T05:12:40.765 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-14T05:12:40.899 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-14T05:12:40.984 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-14T05:12:41.301 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-14T05:12:41.337 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-14T05:12:41.560 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-14T05:12:41.638 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-14T05:12:41.678 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-14T05:12:41.912 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-14T05:12:42.021 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-14T05:12:42.024 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-14T05:12:42.300 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-14T05:12:42.339 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-14T05:12:42.628 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-14T05:12:42.729 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librados2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:42.765 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-14T05:12:42.830 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:42.947 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-14T05:12:43.292 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:43.366 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-14T05:12:43.418 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libradosstriper1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:43.492 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-14T05:12:43.544 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-14T05:12:43.760 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-14T05:12:43.821 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librbd1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:43.947 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-modules-core (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:44.073 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:44.096 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-14T05:12:44.274 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-14T05:12:44.274 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-14T05:12:44.356 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-14T05:12:44.724 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up libcephfs-dev (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:44.724 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-14T05:12:44.850 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rados (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:44.993 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up librgw2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:45.017 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-14T05:12:45.119 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-14T05:12:45.344 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-14T05:12:45.495 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librados2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:45.622 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:45.747 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-14T05:12:46.092 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libcephfs2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.243 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libradosstriper1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.331 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rbd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.369 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-14T05:12:46.448 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up rbd-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.558 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-rgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.645 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librbd1 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.659 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up python3-cephfs (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.771 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-modules-core (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.785 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:46.897 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:47.110 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-14T05:12:47.110 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-14T05:12:47.319 INFO:teuthology.orchestra.run.smithi144.stdout:Adding group ceph....done 2024-09-14T05:12:47.573 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up libcephfs-dev (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:47.700 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rados (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:47.721 INFO:teuthology.orchestra.run.smithi144.stdout:Adding system user ceph....done 2024-09-14T05:12:47.825 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up librgw2 (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:47.889 INFO:teuthology.orchestra.run.smithi144.stdout:Setting system user ceph properties....done 2024-09-14T05:12:47.903 INFO:teuthology.orchestra.run.smithi144.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-14T05:12:47.951 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-14T05:12:48.036 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-14T05:12:48.346 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-14T05:12:49.062 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-test (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.105 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rbd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.188 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up radosgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.231 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up rbd-fuse (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.384 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-rgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.510 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up python3-cephfs (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.627 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-common (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:49.691 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:12:49.691 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:12:50.138 INFO:teuthology.orchestra.run.smithi194.stdout:Adding group ceph....done 2024-09-14T05:12:50.165 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-base (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:50.492 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-14T05:12:50.513 INFO:teuthology.orchestra.run.smithi194.stdout:Adding system user ceph....done 2024-09-14T05:12:50.673 INFO:teuthology.orchestra.run.smithi194.stdout:Setting system user ceph properties....done 2024-09-14T05:12:50.683 INFO:teuthology.orchestra.run.smithi194.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-14T05:12:50.816 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-14T05:12:51.011 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mds (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:51.161 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-14T05:12:51.307 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-14T05:12:51.307 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-14T05:12:51.679 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-test (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:51.744 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:51.891 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up radosgw (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:51.937 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-14T05:12:51.937 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-14T05:12:52.399 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-osd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:52.454 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:12:52.455 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:12:52.667 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-14T05:12:52.667 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-14T05:12:52.898 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-base (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:53.156 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:53.244 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-14T05:12:53.282 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:53.425 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mon (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:53.622 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-14T05:12:53.622 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-14T05:12:53.701 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mds (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-14T05:12:53.944 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-14T05:12:54.151 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:54.285 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:54.386 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-mgr-dashboard (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:54.410 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:54.610 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-14T05:12:54.610 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-14T05:12:54.637 INFO:teuthology.orchestra.run.smithi144.stdout:Setting up ceph-volume (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:55.015 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-14T05:12:55.061 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-osd (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-14T05:12:55.260 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-14T05:12:55.343 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-14T05:12:55.344 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-14T05:12:55.804 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:55.930 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:56.069 INFO:teuthology.orchestra.run.smithi144.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-14T05:12:56.081 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mon (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:56.298 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-14T05:12:56.298 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-14T05:12:56.732 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-cephadm (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:56.739 INFO:teuthology.orchestra.run.smithi144.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-14T05:12:56.745 DEBUG:teuthology.parallel:result is None 2024-09-14T05:12:56.884 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:57.001 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-mgr-dashboard (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:57.127 INFO:teuthology.orchestra.run.smithi194.stdout:Setting up ceph-volume (19.3.0-4775-g7f24e589-1jammy) ... 2024-09-14T05:12:57.463 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-14T05:12:57.572 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-14T05:12:57.725 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-14T05:12:58.617 INFO:teuthology.orchestra.run.smithi194.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-14T05:12:59.270 INFO:teuthology.orchestra.run.smithi194.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-14T05:12:59.275 DEBUG:teuthology.parallel:result is None 2024-09-14T05:12:59.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:12:59.418 DEBUG:teuthology.orchestra.run.smithi144:> dpkg-query -W -f '${Version}' ceph 2024-09-14T05:12:59.437 INFO:teuthology.orchestra.run.smithi144.stdout:19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:12:59.437 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:12:59.438 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775-g7f24e589-1jammy is installed. 2024-09-14T05:12:59.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:12:59.582 DEBUG:teuthology.orchestra.run.smithi194:> dpkg-query -W -f '${Version}' ceph 2024-09-14T05:12:59.602 INFO:teuthology.orchestra.run.smithi194.stdout:19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:12:59.602 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775-g7f24e589-1jammy 2024-09-14T05:12:59.602 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775-g7f24e589-1jammy is installed. 2024-09-14T05:12:59.604 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-14T05:12:59.604 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:12:59.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T05:12:59.642 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:12:59.642 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T05:12:59.661 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-14T05:12:59.661 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:12:59.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T05:12:59.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T05:12:59.756 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:12:59.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T05:12:59.770 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T05:12:59.829 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-14T05:12:59.830 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:12:59.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T05:12:59.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T05:12:59.937 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:12:59.937 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T05:12:59.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T05:13:00.007 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-14T05:13:00.007 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:00.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T05:13:00.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T05:13:00.074 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:00.074 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T05:13:00.087 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T05:13:00.142 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T05:13:00.290 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-14T05:13:00.291 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2024-09-14T05:13:00.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-09-14T05:13:00.306 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-14T05:13:00.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T05:13:00.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T05:13:00.362 INFO:tasks.ceph:Creating extra log directories... 2024-09-14T05:13:00.362 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T05:13:00.404 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T05:13:00.425 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-14T05:13:00.425 INFO:tasks.ceph:config {'conf': {'client': {'rbd_persistent_cache_mode': 'rwl', 'rbd_persistent_cache_path': '/home/ubuntu/cephtest/rbd-pwl-cache', 'rbd_persistent_cache_size': 1073741824, 'rbd_plugins': 'pwl_cache'}, 'global': {'mon warn on pool no app': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T05:13:00.425 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293', 'branch': 'main', 'description': 'rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/recovery conf/{disable-pool-app}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900293', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'rbd_persistent_cache_mode': 'rwl', 'rbd_persistent_cache_path': '/home/ubuntu/cephtest/rbd-pwl-cache', 'rbd_persistent_cache_size': 1073741824, 'rbd_plugins': 'pwl_cache'}, 'global': {'mon warn on pool no app': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, 'workunit': {'branch': 'main', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1'], ['mon.b', 'mgr.y', 'osd.2', 'osd.3', 'client.0']], 'seed': 5839, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'sleep_before_teardown': 0, 'subset': '99/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'targets': {'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8kpNW67uWt3jLixrDmx6tIjTrAhFQ/hooIyMI/wJjjLjQY7XzlaRUtT+gerQCqEaIlEYMYxNpqryTIaa12wx0=', 'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTKoicBGnMGzZsKnv74ZXTdyUe9z6sNsHJOrkGDG1aIKopy0wOIH/fz8ch0qoUc+068+kzqoIOkZhlbkBvzxc0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['mkdir -m 777 /home/ubuntu/cephtest/rbd-pwl-cache']}}, {'exec_on_cleanup': {'client.0': ['rm -rf /home/ubuntu/cephtest/rbd-pwl-cache']}}, {'rbd.create_image': {'client.0': {'image_format': 2, 'image_name': 'testimage', 'image_size': 10240}}}, {'rbd_pwl_cache_recovery': {'client.0': {'image_name': 'testimage'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_20:16:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T05:13:00.426 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-14T05:13:00.460 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-14T05:13:00.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:13:00.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:13:00.522 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:00.522 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T05:13:00.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T05:13:00.567 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-09-14T05:13:00.615 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T05:13:00.615 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:00.615 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-09-14T05:13:00.615 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:00.616 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-14 05:07:32.558065046 +0000 2024-09-14T05:13:00.616 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-14 05:07:32.278067548 +0000 2024-09-14T05:13:00.616 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-14 05:07:32.278067548 +0000 2024-09-14T05:13:00.616 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-14T05:13:00.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T05:13:00.669 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-14T05:13:00.669 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-14T05:13:00.670 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000395304 s, 1.3 MB/s 2024-09-14T05:13:00.671 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T05:13:00.721 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-09-14T05:13:00.767 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T05:13:00.767 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:00.767 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-14T05:13:00.767 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:00.767 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-14 05:07:32.994061152 +0000 2024-09-14T05:13:00.768 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-14 05:07:32.774063117 +0000 2024-09-14T05:13:00.768 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-14 05:07:32.774063117 +0000 2024-09-14T05:13:00.768 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-14T05:13:00.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T05:13:00.822 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-14T05:13:00.822 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-14T05:13:00.822 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00040825 s, 1.3 MB/s 2024-09-14T05:13:00.823 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T05:13:00.873 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 911 Links: 1 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-14 05:07:33.530056363 +0000 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-14 05:07:33.254058829 +0000 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-14 05:07:33.254058829 +0000 2024-09-14T05:13:00.920 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-14T05:13:00.921 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T05:13:00.974 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-14T05:13:00.974 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-14T05:13:00.974 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.00043455 s, 1.2 MB/s 2024-09-14T05:13:00.975 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T05:13:01.025 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-09-14T05:13:01.071 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T05:13:01.071 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:01.071 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-14T05:13:01.072 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:01.072 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-14 05:07:33.746054435 +0000 2024-09-14T05:13:01.072 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-14 05:07:33.742054471 +0000 2024-09-14T05:13:01.072 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-14 05:07:33.742054471 +0000 2024-09-14T05:13:01.072 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2024-09-14T05:13:01.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T05:13:01.125 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-14T05:13:01.126 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-14T05:13:01.126 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000387271 s, 1.3 MB/s 2024-09-14T05:13:01.127 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T05:13:01.177 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:01.177 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:01.177 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T05:13:01.183 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T05:13:01.183 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-09-14T05:13:01.231 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T05:13:01.231 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:01.231 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 867 Links: 1 2024-09-14T05:13:01.231 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:01.231 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-14 05:07:31.844517815 +0000 2024-09-14T05:13:01.232 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-14 05:07:31.576520209 +0000 2024-09-14T05:13:01.232 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-14 05:07:31.576520209 +0000 2024-09-14T05:13:01.232 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-09-14T05:13:01.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T05:13:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-14T05:13:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-14T05:13:01.286 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000471623 s, 1.1 MB/s 2024-09-14T05:13:01.287 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T05:13:01.336 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-09-14T05:13:01.382 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-14 05:07:32.320513563 +0000 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-14 05:07:32.052515957 +0000 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-14 05:07:32.052515957 +0000 2024-09-14T05:13:01.383 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-09-14T05:13:01.383 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T05:13:01.437 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-14T05:13:01.437 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-14T05:13:01.437 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000420334 s, 1.2 MB/s 2024-09-14T05:13:01.439 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T05:13:01.488 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-09-14T05:13:01.534 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-14 05:07:32.752509704 +0000 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-14 05:07:32.528511705 +0000 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-14 05:07:32.528511705 +0000 2024-09-14T05:13:01.535 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-09-14T05:13:01.535 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T05:13:01.589 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-14T05:13:01.589 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-14T05:13:01.589 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00041138 s, 1.2 MB/s 2024-09-14T05:13:01.590 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T05:13:01.640 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-09-14T05:13:01.686 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-14 05:07:33.236505381 +0000 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-14 05:07:32.980507667 +0000 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-14 05:07:32.980507667 +0000 2024-09-14T05:13:01.687 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-09-14T05:13:01.687 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T05:13:01.741 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-14T05:13:01.741 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-14T05:13:01.741 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000381866 s, 1.3 MB/s 2024-09-14T05:13:01.742 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T05:13:01.792 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:01.792 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-14T05:13:01.792 INFO:tasks.ceph:Generating config... 2024-09-14T05:13:01.793 INFO:tasks.ceph:[client] rbd_persistent_cache_mode = rwl 2024-09-14T05:13:01.794 INFO:tasks.ceph:[client] rbd_persistent_cache_path = /home/ubuntu/cephtest/rbd-pwl-cache 2024-09-14T05:13:01.794 INFO:tasks.ceph:[client] rbd_persistent_cache_size = 1073741824 2024-09-14T05:13:01.794 INFO:tasks.ceph:[client] rbd_plugins = pwl_cache 2024-09-14T05:13:01.794 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T05:13:01.794 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T05:13:01.794 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T05:13:01.794 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T05:13:01.794 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T05:13:01.794 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T05:13:01.794 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T05:13:01.794 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T05:13:01.794 INFO:tasks.ceph:Setting up mon.a... 2024-09-14T05:13:01.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-14T05:13:01.828 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.keyring 2024-09-14T05:13:01.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-14T05:13:01.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-14T05:13:01.971 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.144'), ('mon.b', '172.21.15.194')] 2024-09-14T05:13:01.972 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.144,172.21.15.194', 'mon warn on pool no app': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd_persistent_cache_mode': 'rwl', 'rbd_persistent_cache_path': '/home/ubuntu/cephtest/rbd-pwl-cache', 'rbd_persistent_cache_size': 1073741824, 'rbd_plugins': 'pwl_cache'}, 'mon.a': {}, 'mon.b': {}} 2024-09-14T05:13:01.972 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:01.972 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:13:02.019 DEBUG:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.144 --add b 172.21.15.194 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: generated fsid aaa809bf-6549-4b19-b0ac-70959376c827 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:setting min_mon_release = quincy 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:epoch 0 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:fsid aaa809bf-6549-4b19-b0ac-70959376c827 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:last_changed 2024-09-14T05:13:02.105941+0000 2024-09-14T05:13:02.105 INFO:teuthology.orchestra.run.smithi144.stdout:created 2024-09-14T05:13:02.105941+0000 2024-09-14T05:13:02.106 INFO:teuthology.orchestra.run.smithi144.stdout:min_mon_release 17 (quincy) 2024-09-14T05:13:02.106 INFO:teuthology.orchestra.run.smithi144.stdout:election_strategy: 1 2024-09-14T05:13:02.106 INFO:teuthology.orchestra.run.smithi144.stdout:0: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.a 2024-09-14T05:13:02.106 INFO:teuthology.orchestra.run.smithi144.stdout:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-09-14T05:13:02.106 INFO:teuthology.orchestra.run.smithi144.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (2 monitors) 2024-09-14T05:13:02.107 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:13:02.155 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID aaa809bf-6549-4b19-b0ac-70959376c827... 2024-09-14T05:13:02.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-14T05:13:02.200 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-14T05:13:02.227 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2024-09-14T05:13:02.227 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2024-09-14T05:13:02.227 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:13:02.227 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:13:02.228 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:13:02.229 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2024-09-14T05:13:02.230 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.144,172.21.15.194" 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on pool no app = False 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = aaa809bf-6549-4b19-b0ac-70959376c827 2024-09-14T05:13:02.231 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2024-09-14T05:13:02.232 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug trim objects = true 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.233 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.234 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:13:02.235 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2024-09-14T05:13:02.236 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: rbd_persistent_cache_mode = rwl 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: rbd_persistent_cache_path = /home/ubuntu/cephtest/rbd-pwl-cache 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: rbd_persistent_cache_size = 1073741824 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout: rbd_plugins = pwl_cache 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2024-09-14T05:13:02.237 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-09-14T05:13:02.238 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-09-14T05:13:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-09-14T05:13:02.240 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "172.21.15.144,172.21.15.194" 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on pool no app = False 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = aaa809bf-6549-4b19-b0ac-70959376c827 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.241 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug trim objects = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.242 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-09-14T05:13:02.243 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-09-14T05:13:02.244 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:13:02.245 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rbd_persistent_cache_mode = rwl 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rbd_persistent_cache_path = /home/ubuntu/cephtest/rbd-pwl-cache 2024-09-14T05:13:02.246 INFO:teuthology.orchestra.run.smithi194.stdout: rbd_persistent_cache_size = 1073741824 2024-09-14T05:13:02.247 INFO:teuthology.orchestra.run.smithi194.stdout: rbd_plugins = pwl_cache 2024-09-14T05:13:02.247 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-09-14T05:13:02.247 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-09-14T05:13:02.248 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-14T05:13:02.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-14T05:13:02.339 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T05:13:02.340 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:02.340 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-14T05:13:02.345 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:02.345 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-14T05:13:02.391 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:13:02.392 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:02.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-14T05:13:02.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-14T05:13:02.455 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:02.455 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:02.504 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:13:02.504 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:02.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-14T05:13:02.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-14T05:13:02.527 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:02.527 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:02.575 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T05:13:02.575 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T05:13:02.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-14T05:13:02.619 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-14T05:13:02.622 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-14T05:13:02.664 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-14T05:13:02.673 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T05:13:02.673 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T05:13:02.674 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-14T05:13:02.745 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-14T05:13:02.757 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T05:13:02.757 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-09-14T05:13:02.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-14T05:13:02.772 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:02.772 INFO:tasks.ceph:role: osd.0 2024-09-14T05:13:02.772 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:13:02.772 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T05:13:02.834 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:13:02.835 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:13:03.294 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-14T05:13:03.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-14T05:13:03.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-14T05:13:03.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-14T05:13:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:03.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-14T05:13:03.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-14T05:13:03.453 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:03.453 INFO:tasks.ceph:role: osd.1 2024-09-14T05:13:03.453 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:13:03.453 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T05:13:03.515 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:13:03.515 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:13:03.516 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:13:03.981 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-09-14T05:13:03.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-09-14T05:13:03.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-14T05:13:04.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-14T05:13:04.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: /sbin/restorecon: command not found 2024-09-14T05:13:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:04.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:04.260 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:04.257+0000 7fd694e1d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-14T05:13:04.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:04.261+0000 7fd694e1d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-14T05:13:04.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:04.261+0000 7fd694e1d7c0 -1 bdev(0x5649222ed800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-14T05:13:04.267 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:04.265+0000 7fd694e1d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-14T05:13:06.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-14T05:13:06.577 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:06.657 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:06.652+0000 7faa1f58d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-14T05:13:06.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:06.652+0000 7faa1f58d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-14T05:13:06.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:06.652+0000 7faa1f58d7c0 -1 bdev(0x55dc480b1800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-14T05:13:06.658 INFO:teuthology.orchestra.run.smithi144.stderr:2024-09-14T05:13:06.652+0000 7faa1f58d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-14T05:13:08.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-14T05:13:09.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-14T05:13:09.030 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:09.030 INFO:tasks.ceph:role: osd.2 2024-09-14T05:13:09.030 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:13:09.030 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T05:13:09.095 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:13:09.095 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:13:09.095 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:13:09.095 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:13:09.096 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:13:09.584 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-14T05:13:09.586 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-14T05:13:09.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-14T05:13:09.742 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-14T05:13:09.754 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: /sbin/restorecon: command not found 2024-09-14T05:13:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:09.755 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-14T05:13:09.811 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-09-14T05:13:09.811 INFO:tasks.ceph:role: osd.3 2024-09-14T05:13:09.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:13:09.812 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:13:09.875 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:13:10.314 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-09-14T05:13:10.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-09-14T05:13:10.317 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-09-14T05:13:10.405 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-14T05:13:10.413 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: /sbin/restorecon: command not found 2024-09-14T05:13:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:10.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:10.523 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:10.522+0000 7f93a668f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-14T05:13:10.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:10.522+0000 7f93a668f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-14T05:13:10.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:10.522+0000 7f93a668f7c0 -1 bdev(0x561c0f4b1800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-14T05:13:10.525 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:10.522+0000 7f93a668f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-14T05:13:12.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-14T05:13:12.841 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:12.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:12.918+0000 7fb32df6e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-14T05:13:12.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:12.918+0000 7fb32df6e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-14T05:13:12.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:12.918+0000 7fb32df6e7c0 -1 bdev(0x560f92489800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-14T05:13:12.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-14T05:13:12.918+0000 7fb32df6e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-14T05:13:15.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-14T05:13:15.266 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T05:13:15.266 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:15.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-14T05:13:15.281 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:15.281 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-14T05:13:15.338 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:15.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-14T05:13:15.400 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:15.401 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-14T05:13:15.414 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:15.415 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-14T05:13:15.475 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:15.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-14T05:13:15.540 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:15.540 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-14T05:13:15.595 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T05:13:15.595 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-14T05:13:15.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-14T05:13:15.644 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBeG+VmHDb7JBAAtZ8HWp96YfmqoizeUAabgg== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBgG+VmbeePDxAAbOpNEwKotuP9hnR1shSbEQ== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBiG+Vm1tlHJxAA0sjNuHVle/brEopyPhrRdg== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.y] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBeG+VmORmtJxAA1uY+ozfpj9e616CKkimJLg== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBmG+VmBhdOHxAAy8RAD4IHmfzyucekvu+J6g== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBoG+Vm6zr0NhAAF1Tx+pvFuD3x6pxTFZpoNg== 2024-09-14T05:13:15.645 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2024-09-14T05:13:15.646 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBeG+Vme1aMLBAASM+kZrVK++QjzAeZJHSgbg== 2024-09-14T05:13:15.650 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBeG+VmHDb7JBAAtZ8HWp96YfmqoizeUAabgg== 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBgG+VmbeePDxAAbOpNEwKotuP9hnR1shSbEQ== 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBiG+Vm1tlHJxAA0sjNuHVle/brEopyPhrRdg== 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBeG+VmORmtJxAA1uY+ozfpj9e616CKkimJLg== 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBmG+VmBhdOHxAAy8RAD4IHmfzyucekvu+J6g== 2024-09-14T05:13:15.651 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-09-14T05:13:15.652 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBoG+Vm6zr0NhAAF1Tx+pvFuD3x6pxTFZpoNg== 2024-09-14T05:13:15.652 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2024-09-14T05:13:15.652 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBeG+Vme1aMLBAASM+kZrVK++QjzAeZJHSgbg== 2024-09-14T05:13:15.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:13:15.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:13:15.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:15.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:15.829 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:15.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:15.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:13:15.916 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:13:15.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:16.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:16.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:16.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:13:16.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:13:16.169 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:13:16.256 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T05:13:16.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-14T05:13:16.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-14T05:13:16.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-14T05:13:16.376 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-14T05:13:16.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-14T05:13:16.483 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-14T05:13:16.493 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:16.495 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-14T05:13:16.538 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-14T05:13:16.538 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-14T05:13:16.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-14T05:13:16.544 INFO:tasks.ceph.mon.a:Started 2024-09-14T05:13:16.544 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-14T05:13:16.544 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-14T05:13:16.583 INFO:tasks.ceph.mon.b:Started 2024-09-14T05:13:16.584 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-14T05:13:16.584 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-14T05:13:16.584 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-14T05:13:16.586 INFO:tasks.ceph.mgr.x:Started 2024-09-14T05:13:16.587 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-14T05:13:16.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-14T05:13:16.589 INFO:tasks.ceph.mgr.y:Started 2024-09-14T05:13:16.589 DEBUG:tasks.ceph:set 0 configs 2024-09-14T05:13:16.589 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T05:13:16.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd crush tunables default 2024-09-14T05:13:17.006 INFO:teuthology.orchestra.run.smithi144.stderr:adjusted tunables profile to default 2024-09-14T05:13:17.025 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T05:13:17.025 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-14T05:13:17.025 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:17.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-14T05:13:17.036 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-14T05:13:17.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-14T05:13:17.093 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:17.093 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-14T05:13:17.108 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-14T05:13:17.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-14T05:13:17.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new d8387a05-b5c5-4510-baa7-ae9d47d0fde6 0 2024-09-14T05:13:17.438 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-09-14T05:13:17.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 0908a852-a99b-4d23-ae6f-65ed6b425bfc 1 2024-09-14T05:13:17.661 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-09-14T05:13:17.679 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new c0ce7c85-9863-4713-8659-aeabca49bbcd 2 2024-09-14T05:13:17.887 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-09-14T05:13:17.903 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f 3 2024-09-14T05:13:18.107 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-09-14T05:13:18.124 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-14T05:13:18.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-14T05:13:18.126 INFO:tasks.ceph.osd.0:Started 2024-09-14T05:13:18.126 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-14T05:13:18.126 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-14T05:13:18.128 INFO:tasks.ceph.osd.1:Started 2024-09-14T05:13:18.128 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-14T05:13:18.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-14T05:13:18.130 INFO:tasks.ceph.osd.2:Started 2024-09-14T05:13:18.130 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-14T05:13:18.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-14T05:13:18.132 INFO:tasks.ceph.osd.3:Started 2024-09-14T05:13:18.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:18.311 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:18.312 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":6,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:18.100075+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:18.329 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-14T05:13:18.329 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T05:13:18.732 DEBUG:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:18.916 INFO:teuthology.misc.health.smithi144.stdout: 2024-09-14T05:13:18.916 INFO:teuthology.misc.health.smithi144.stdout:{"epoch":6,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:18.100075+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:18.932 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-14T05:13:19.279 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-09-14T05:13:19.278+0000 7f89013617c0 -1 Falling back to public interface 2024-09-14T05:13:19.552 INFO:tasks.ceph.osd.0.smithi144.stderr:2024-09-14T05:13:19.548+0000 7ff586e8b7c0 -1 Falling back to public interface 2024-09-14T05:13:19.713 INFO:tasks.ceph.mgr.x.smithi144.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-14T05:13:19.714 INFO:tasks.ceph.mgr.x.smithi144.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-14T05:13:19.714 INFO:tasks.ceph.mgr.x.smithi144.stderr: from numpy import show_config as show_numpy_config 2024-09-14T05:13:19.748 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-09-14T05:13:19.746+0000 7f61cf25d7c0 -1 Falling back to public interface 2024-09-14T05:13:19.780 INFO:tasks.ceph.osd.1.smithi144.stderr:2024-09-14T05:13:19.776+0000 7f84282327c0 -1 Falling back to public interface 2024-09-14T05:13:19.917 INFO:tasks.ceph.mgr.y.smithi194.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-14T05:13:19.917 INFO:tasks.ceph.mgr.y.smithi194.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-14T05:13:19.917 INFO:tasks.ceph.mgr.y.smithi194.stderr: from numpy import show_config as show_numpy_config 2024-09-14T05:13:22.645 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-09-14T05:13:22.642+0000 7f89013617c0 -1 osd.2 0 log_to_monitors true 2024-09-14T05:13:22.700 INFO:tasks.ceph.osd.0.smithi144.stderr:2024-09-14T05:13:22.696+0000 7ff586e8b7c0 -1 osd.0 0 log_to_monitors true 2024-09-14T05:13:22.710 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-09-14T05:13:22.704+0000 7f5f60bf0640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-14T05:13:22.822 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-09-14T05:13:22.818+0000 7f61cf25d7c0 -1 osd.3 0 log_to_monitors true 2024-09-14T05:13:22.944 INFO:tasks.ceph.osd.1.smithi144.stderr:2024-09-14T05:13:22.940+0000 7f84282327c0 -1 osd.1 0 log_to_monitors true 2024-09-14T05:13:24.373 INFO:tasks.ceph.osd.0.smithi144.stderr:2024-09-14T05:13:24.368+0000 7ff583646640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T05:13:24.392 INFO:tasks.ceph.osd.0.smithi144.stderr:2024-09-14T05:13:24.388+0000 7ff57ac66640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:13:24.419 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-09-14T05:13:24.418+0000 7f88fdb1c640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T05:13:24.439 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-09-14T05:13:24.438+0000 7f88f513c640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:13:25.336 DEBUG:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:25.499 INFO:tasks.ceph.osd.1.smithi144.stderr:2024-09-14T05:13:25.496+0000 7f84249ed640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T05:13:25.523 INFO:tasks.ceph.osd.1.smithi144.stderr:2024-09-14T05:13:25.520+0000 7f841c00d640 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:13:25.643 INFO:teuthology.misc.health.smithi144.stdout: 2024-09-14T05:13:25.643 INFO:teuthology.misc.health.smithi144.stdout:{"epoch":9,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:24.759801+0000","last_up_change":"2024-09-14T05:13:24.759801+0000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6801","nonce":736494405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6803","nonce":736494405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6807","nonce":736494405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6805","nonce":736494405}]},"public_addr":"172.21.15.144:6801/736494405","cluster_addr":"172.21.15.144:6803/736494405","heartbeat_back_addr":"172.21.15.144:6807/736494405","heartbeat_front_addr":"172.21.15.144:6805/736494405","state":["exists","up"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6801","nonce":520007229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6803","nonce":520007229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6807","nonce":520007229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6805","nonce":520007229}]},"public_addr":"172.21.15.194:6801/520007229","cluster_addr":"172.21.15.194:6803/520007229","heartbeat_back_addr":"172.21.15.194:6807/520007229","heartbeat_front_addr":"172.21.15.194:6805/520007229","state":["exists","up"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:25.660 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-09-14T05:13:25.658+0000 7f61cb206640 -1 osd.3 0 waiting for initial osdmap 2024-09-14T05:13:25.661 DEBUG:teuthology.misc:2 of 4 OSDs are up 2024-09-14T05:13:25.687 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-09-14T05:13:25.686+0000 7f61c3038640 -1 osd.3 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:13:32.065 DEBUG:teuthology.orchestra.run.smithi144:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:32.383 INFO:teuthology.misc.health.smithi144.stdout: 2024-09-14T05:13:32.383 INFO:teuthology.misc.health.smithi144.stdout:{"epoch":12,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:28.509553+0000","last_up_change":"2024-09-14T05:13:26.503913+0000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:13:25.740988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6801","nonce":736494405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6803","nonce":736494405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6807","nonce":736494405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6805","nonce":736494405}]},"public_addr":"172.21.15.144:6801/736494405","cluster_addr":"172.21.15.144:6803/736494405","heartbeat_back_addr":"172.21.15.144:6807/736494405","heartbeat_front_addr":"172.21.15.144:6805/736494405","state":["exists","up"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6809","nonce":2747334951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6811","nonce":2747334951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6815","nonce":2747334951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6813","nonce":2747334951}]},"public_addr":"172.21.15.144:6809/2747334951","cluster_addr":"172.21.15.144:6811/2747334951","heartbeat_back_addr":"172.21.15.144:6815/2747334951","heartbeat_front_addr":"172.21.15.144:6813/2747334951","state":["exists","up"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6801","nonce":520007229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6803","nonce":520007229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6807","nonce":520007229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6805","nonce":520007229}]},"public_addr":"172.21.15.194:6801/520007229","cluster_addr":"172.21.15.194:6803/520007229","heartbeat_back_addr":"172.21.15.194:6807/520007229","heartbeat_front_addr":"172.21.15.194:6805/520007229","state":["exists","up"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6809","nonce":796375837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6811","nonce":796375837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6815","nonce":796375837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6813","nonce":796375837}]},"public_addr":"172.21.15.194:6809/796375837","cluster_addr":"172.21.15.194:6811/796375837","heartbeat_back_addr":"172.21.15.194:6815/796375837","heartbeat_front_addr":"172.21.15.194:6813/796375837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.664216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.993903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.627468+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.788647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:32.402 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-14T05:13:32.402 INFO:tasks.ceph:Creating RBD pool 2024-09-14T05:13:32.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-14T05:13:33.551 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'rbd' created 2024-09-14T05:13:33.577 DEBUG:teuthology.orchestra.run.smithi144:> rbd --cluster ceph pool init rbd 2024-09-14T05:13:36.603 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-14T05:13:36.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-14T05:13:36.605 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T05:13:36.917 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:36.935 INFO:teuthology.orchestra.run.smithi144.stdout:[{"version":5,"timestamp":"2024-09-14T05:13:25.662307+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"8327.946656"}]}] 2024-09-14T05:13:36.935 INFO:tasks.ceph_manager:config epoch is 5 2024-09-14T05:13:36.935 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-14T05:13:36.936 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-14T05:13:36.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-14T05:13:37.283 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:37.301 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":387272432},{"type":"v1","addr":"172.21.15.144:6817","nonce":387272432}]},"active_addr":"172.21.15.144:6817/387272432","active_change":"2024-09-14T05:13:21.690173+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3084852199}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3024577904}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3086384793}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":470294783}]}]} 2024-09-14T05:13:37.305 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-14T05:13:37.305 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-14T05:13:37.305 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:37.586 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:37.586 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":16,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:36.571988+0000","last_up_change":"2024-09-14T05:13:26.503913+0000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:13:25.740988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:13:32.715764+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6801","nonce":736494405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6803","nonce":736494405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6807","nonce":736494405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6805","nonce":736494405}]},"public_addr":"172.21.15.144:6801/736494405","cluster_addr":"172.21.15.144:6803/736494405","heartbeat_back_addr":"172.21.15.144:6807/736494405","heartbeat_front_addr":"172.21.15.144:6805/736494405","state":["exists","up"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6809","nonce":2747334951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6811","nonce":2747334951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6815","nonce":2747334951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6813","nonce":2747334951}]},"public_addr":"172.21.15.144:6809/2747334951","cluster_addr":"172.21.15.144:6811/2747334951","heartbeat_back_addr":"172.21.15.144:6815/2747334951","heartbeat_front_addr":"172.21.15.144:6813/2747334951","state":["exists","up"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6801","nonce":520007229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6803","nonce":520007229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6807","nonce":520007229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6805","nonce":520007229}]},"public_addr":"172.21.15.194:6801/520007229","cluster_addr":"172.21.15.194:6803/520007229","heartbeat_back_addr":"172.21.15.194:6807/520007229","heartbeat_front_addr":"172.21.15.194:6805/520007229","state":["exists","up"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6809","nonce":796375837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6811","nonce":796375837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6815","nonce":796375837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6813","nonce":796375837}]},"public_addr":"172.21.15.194:6809/796375837","cluster_addr":"172.21.15.194:6811/796375837","heartbeat_back_addr":"172.21.15.194:6815/796375837","heartbeat_front_addr":"172.21.15.194:6813/796375837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.664216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.993903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.627468+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.788647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:37.604 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-14T05:13:37.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-14T05:13:37.885 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:37.885 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":16,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:13:36.571988+0000","last_up_change":"2024-09-14T05:13:26.503913+0000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:13:25.740988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:13:32.715764+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6801","nonce":736494405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6803","nonce":736494405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6807","nonce":736494405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6805","nonce":736494405}]},"public_addr":"172.21.15.144:6801/736494405","cluster_addr":"172.21.15.144:6803/736494405","heartbeat_back_addr":"172.21.15.144:6807/736494405","heartbeat_front_addr":"172.21.15.144:6805/736494405","state":["exists","up"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6809","nonce":2747334951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6811","nonce":2747334951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6815","nonce":2747334951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6813","nonce":2747334951}]},"public_addr":"172.21.15.144:6809/2747334951","cluster_addr":"172.21.15.144:6811/2747334951","heartbeat_back_addr":"172.21.15.144:6815/2747334951","heartbeat_front_addr":"172.21.15.144:6813/2747334951","state":["exists","up"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6801","nonce":520007229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6803","nonce":520007229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6807","nonce":520007229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6805","nonce":520007229}]},"public_addr":"172.21.15.194:6801/520007229","cluster_addr":"172.21.15.194:6803/520007229","heartbeat_back_addr":"172.21.15.194:6807/520007229","heartbeat_front_addr":"172.21.15.194:6805/520007229","state":["exists","up"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6809","nonce":796375837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6811","nonce":796375837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6815","nonce":796375837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6813","nonce":796375837}]},"public_addr":"172.21.15.194:6809/796375837","cluster_addr":"172.21.15.194:6811/796375837","heartbeat_back_addr":"172.21.15.194:6815/796375837","heartbeat_front_addr":"172.21.15.194:6813/796375837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.664216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.993903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.627468+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.788647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:13:37.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-14T05:13:37.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-14T05:13:37.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-14T05:13:37.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-14T05:13:38.059 INFO:teuthology.orchestra.run.smithi144.stdout:38654705669 2024-09-14T05:13:38.059 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-14T05:13:38.062 INFO:teuthology.orchestra.run.smithi144.stdout:38654705668 2024-09-14T05:13:38.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-14T05:13:38.072 INFO:teuthology.orchestra.run.smithi144.stdout:42949672965 2024-09-14T05:13:38.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-14T05:13:38.074 INFO:teuthology.orchestra.run.smithi144.stdout:42949672965 2024-09-14T05:13:38.074 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-14T05:13:38.357 INFO:teuthology.orchestra.run.smithi144.stdout:38654705668 2024-09-14T05:13:38.367 INFO:teuthology.orchestra.run.smithi144.stdout:38654705667 2024-09-14T05:13:38.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.0 2024-09-14T05:13:38.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2024-09-14T05:13:38.405 INFO:teuthology.orchestra.run.smithi144.stdout:42949672963 2024-09-14T05:13:38.415 INFO:teuthology.orchestra.run.smithi144.stdout:42949672963 2024-09-14T05:13:38.423 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672963 for osd.3 2024-09-14T05:13:38.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672963 for osd.1 2024-09-14T05:13:39.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-14T05:13:39.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-14T05:13:39.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-14T05:13:39.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-14T05:13:39.698 INFO:teuthology.orchestra.run.smithi144.stdout:38654705668 2024-09-14T05:13:39.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.0 2024-09-14T05:13:39.729 INFO:teuthology.orchestra.run.smithi144.stdout:42949672964 2024-09-14T05:13:39.741 INFO:teuthology.orchestra.run.smithi144.stdout:38654705667 2024-09-14T05:13:39.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672964 for osd.3 2024-09-14T05:13:39.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2024-09-14T05:13:39.762 INFO:teuthology.orchestra.run.smithi144.stdout:42949672964 2024-09-14T05:13:39.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672964 for osd.1 2024-09-14T05:13:40.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-14T05:13:40.748 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-14T05:13:40.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-14T05:13:40.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-14T05:13:41.086 INFO:teuthology.orchestra.run.smithi144.stdout:42949672965 2024-09-14T05:13:41.088 INFO:teuthology.orchestra.run.smithi144.stdout:38654705669 2024-09-14T05:13:41.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.3 2024-09-14T05:13:41.108 DEBUG:teuthology.parallel:result is None 2024-09-14T05:13:41.108 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.0 2024-09-14T05:13:41.108 DEBUG:teuthology.parallel:result is None 2024-09-14T05:13:41.119 INFO:teuthology.orchestra.run.smithi144.stdout:38654705668 2024-09-14T05:13:41.130 INFO:teuthology.orchestra.run.smithi144.stdout:42949672965 2024-09-14T05:13:41.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-09-14T05:13:41.138 DEBUG:teuthology.parallel:result is None 2024-09-14T05:13:41.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.1 2024-09-14T05:13:41.148 DEBUG:teuthology.parallel:result is None 2024-09-14T05:13:41.148 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-14T05:13:41.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:13:41.468 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:41.468 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:13:41.485 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T05:13:39.702635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":109848,"kb_used_data":1712,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":419320552,"statfs":{"total":429496729600,"available":429384245248,"internally_reserved":0,"allocated":1753088,"data_stored":1361670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001179"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.589135+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:13:36.589135+0000","last_peered":"2024-09-14T05:13:36.589135+0000","last_clean":"2024-09-14T05:13:36.589135+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:13:36.589135+0000","last_undegraded":"2024-09-14T05:13:36.589135+0000","last_fullsized":"2024-09-14T05:13:36.589135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.583298+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:13:36.583298+0000","last_peered":"2024-09-14T05:13:36.583298+0000","last_clean":"2024-09-14T05:13:36.583298+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:13:36.583298+0000","last_undegraded":"2024-09-14T05:13:36.583298+0000","last_fullsized":"2024-09-14T05:13:36.583298+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.588684+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:13:36.588684+0000","last_peered":"2024-09-14T05:13:36.588684+0000","last_clean":"2024-09-14T05:13:36.588684+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:13:36.588684+0000","last_undegraded":"2024-09-14T05:13:36.588684+0000","last_fullsized":"2024-09-14T05:13:36.588684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.582987+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:13:36.582987+0000","last_peered":"2024-09-14T05:13:36.582987+0000","last_clean":"2024-09-14T05:13:36.582987+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:13:36.582987+0000","last_undegraded":"2024-09-14T05:13:36.582987+0000","last_fullsized":"2024-09-14T05:13:36.582987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.591069+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:13:36.591069+0000","last_peered":"2024-09-14T05:13:36.591069+0000","last_clean":"2024-09-14T05:13:36.591069+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:13:36.591069+0000","last_undegraded":"2024-09-14T05:13:36.591069+0000","last_fullsized":"2024-09-14T05:13:36.591069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.582721+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:13:36.582721+0000","last_peered":"2024-09-14T05:13:36.582721+0000","last_clean":"2024-09-14T05:13:36.582721+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:13:36.582721+0000","last_undegraded":"2024-09-14T05:13:36.582721+0000","last_fullsized":"2024-09-14T05:13:36.582721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.588563+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:13:36.588563+0000","last_peered":"2024-09-14T05:13:36.588563+0000","last_clean":"2024-09-14T05:13:36.588563+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:13:36.588563+0000","last_undegraded":"2024-09-14T05:13:36.588563+0000","last_fullsized":"2024-09-14T05:13:36.588563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.589331+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:13:36.589331+0000","last_peered":"2024-09-14T05:13:36.589331+0000","last_clean":"2024-09-14T05:13:36.589331+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:13:36.589331+0000","last_undegraded":"2024-09-14T05:13:36.589331+0000","last_fullsized":"2024-09-14T05:13:36.589331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'72","reported_seq":111,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.587470+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:13:36.587470+0000","last_peered":"2024-09-14T05:13:36.587470+0000","last_clean":"2024-09-14T05:13:36.587470+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:13:36.587470+0000","last_undegraded":"2024-09-14T05:13:36.587470+0000","last_fullsized":"2024-09-14T05:13:36.587470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27752,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829848,"statfs":{"total":107374182400,"available":107345764352,"internally_reserved":0,"allocated":737280,"data_stored":635611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830424,"statfs":{"total":107374182400,"available":107346354176,"internally_reserved":0,"allocated":143360,"data_stored":45243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830428,"statfs":{"total":107374182400,"available":107346358272,"internally_reserved":0,"allocated":139264,"data_stored":45224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27748,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829852,"statfs":{"total":107374182400,"available":107345768448,"internally_reserved":0,"allocated":733184,"data_stored":635592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:13:41.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:13:41.761 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:41.761 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:13:41.778 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-14T05:13:41.703003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":109848,"kb_used_data":1712,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":419320552,"statfs":{"total":429496729600,"available":429384245248,"internally_reserved":0,"allocated":1753088,"data_stored":1361670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.129935"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.589135+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:13:36.589135+0000","last_peered":"2024-09-14T05:13:36.589135+0000","last_clean":"2024-09-14T05:13:36.589135+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:13:36.589135+0000","last_undegraded":"2024-09-14T05:13:36.589135+0000","last_fullsized":"2024-09-14T05:13:36.589135+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.583298+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:13:36.583298+0000","last_peered":"2024-09-14T05:13:36.583298+0000","last_clean":"2024-09-14T05:13:36.583298+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:13:36.583298+0000","last_undegraded":"2024-09-14T05:13:36.583298+0000","last_fullsized":"2024-09-14T05:13:36.583298+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.588684+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:13:36.588684+0000","last_peered":"2024-09-14T05:13:36.588684+0000","last_clean":"2024-09-14T05:13:36.588684+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:13:36.588684+0000","last_undegraded":"2024-09-14T05:13:36.588684+0000","last_fullsized":"2024-09-14T05:13:36.588684+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.582987+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:13:36.582987+0000","last_peered":"2024-09-14T05:13:36.582987+0000","last_clean":"2024-09-14T05:13:36.582987+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:13:36.582987+0000","last_undegraded":"2024-09-14T05:13:36.582987+0000","last_fullsized":"2024-09-14T05:13:36.582987+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.591069+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:13:36.591069+0000","last_peered":"2024-09-14T05:13:36.591069+0000","last_clean":"2024-09-14T05:13:36.591069+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:13:36.591069+0000","last_undegraded":"2024-09-14T05:13:36.591069+0000","last_fullsized":"2024-09-14T05:13:36.591069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.582721+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:13:36.582721+0000","last_peered":"2024-09-14T05:13:36.582721+0000","last_clean":"2024-09-14T05:13:36.582721+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:13:36.582721+0000","last_undegraded":"2024-09-14T05:13:36.582721+0000","last_fullsized":"2024-09-14T05:13:36.582721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.588563+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:13:36.588563+0000","last_peered":"2024-09-14T05:13:36.588563+0000","last_clean":"2024-09-14T05:13:36.588563+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:13:36.588563+0000","last_undegraded":"2024-09-14T05:13:36.588563+0000","last_fullsized":"2024-09-14T05:13:36.588563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.589331+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:13:36.589331+0000","last_peered":"2024-09-14T05:13:36.589331+0000","last_clean":"2024-09-14T05:13:36.589331+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:13:36.589331+0000","last_undegraded":"2024-09-14T05:13:36.589331+0000","last_fullsized":"2024-09-14T05:13:36.589331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'72","reported_seq":111,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:13:36.587470+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:13:36.587470+0000","last_peered":"2024-09-14T05:13:36.587470+0000","last_clean":"2024-09-14T05:13:36.587470+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:13:36.587470+0000","last_undegraded":"2024-09-14T05:13:36.587470+0000","last_fullsized":"2024-09-14T05:13:36.587470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27752,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829848,"statfs":{"total":107374182400,"available":107345764352,"internally_reserved":0,"allocated":737280,"data_stored":635611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830424,"statfs":{"total":107374182400,"available":107346354176,"internally_reserved":0,"allocated":143360,"data_stored":45243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27172,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830428,"statfs":{"total":107374182400,"available":107346358272,"internally_reserved":0,"allocated":139264,"data_stored":45224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27748,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829852,"statfs":{"total":107374182400,"available":107345768448,"internally_reserved":0,"allocated":733184,"data_stored":635592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:13:41.779 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-14T05:13:41.779 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-14T05:13:41.779 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-14T05:13:41.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-14T05:13:42.084 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:13:42.085 INFO:teuthology.orchestra.run.smithi144.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T05:13:42.102 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-14T05:13:42.102 INFO:teuthology.run_tasks:Running task exec... 2024-09-14T05:13:42.115 INFO:teuthology.task.exec:Executing custom commands... 2024-09-14T05:13:42.115 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:13:42.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir -m 777 /home/ubuntu/cephtest/rbd-pwl-cache' 2024-09-14T05:13:42.130 INFO:teuthology.run_tasks:Running task exec_on_cleanup... 2024-09-14T05:13:42.157 INFO:teuthology.run_tasks:Running task rbd.create_image... 2024-09-14T05:13:42.171 INFO:tasks.rbd:Creating image testimage with size 10240 2024-09-14T05:13:42.172 DEBUG:teuthology.orchestra.run.smithi194:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd --cluster ceph -p rbd create --size 10240 testimage --image-format 2 2024-09-14T05:13:42.303 INFO:teuthology.run_tasks:Running task rbd_pwl_cache_recovery... 2024-09-14T05:13:42.342 INFO:tasks.rbd_pwl_cache_recovery:thrashes rbd bench on persistent write log cache 2024-09-14T05:13:42.342 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:13:42.342 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 16K --io-total 100G testimage 2024-09-14T05:13:43.466 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:13:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:13:45.108 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 51.7246 828 KiB/s 2024-09-14T05:13:45.557 INFO:teuthology.orchestra.run.smithi194.stdout: 2 48 46.3772 742 KiB/s 2024-09-14T05:13:46.518 INFO:teuthology.orchestra.run.smithi194.stdout: 3 80 41.026 656 KiB/s 2024-09-14T05:13:47.611 INFO:teuthology.orchestra.run.smithi194.stdout: 4 112 37.2964 597 KiB/s 2024-09-14T05:13:48.917 INFO:teuthology.orchestra.run.smithi194.stdout: 5 160 37.1625 595 KiB/s 2024-09-14T05:13:49.677 INFO:teuthology.orchestra.run.smithi194.stdout: 6 176 31.4963 504 KiB/s 2024-09-14T05:13:50.644 INFO:teuthology.orchestra.run.smithi194.stdout: 7 208 31.4716 504 KiB/s 2024-09-14T05:13:51.636 INFO:teuthology.orchestra.run.smithi194.stdout: 8 240 31.2747 500 KiB/s 2024-09-14T05:13:52.345 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:13:52.345 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:13:52.402 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3136032906","client":4263,"cookie":140589714673344}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:13:48 2024","present":true,"empty":false,"clean":false,"allocated_bytes":2539520,"cached_bytes":2539520,"dirty_bytes":2539520,"free_bytes":749079756,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:13:52.404 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:13:52.516 INFO:teuthology.orchestra.run.smithi194.stdout: 9 272 32.6267 522 KiB/s 2024-09-14T05:13:53.556 INFO:teuthology.orchestra.run.smithi194.stdout: 10 304 31.0348 497 KiB/s 2024-09-14T05:13:54.527 INFO:teuthology.orchestra.run.smithi194.stdout: 11 336 33.0036 528 KiB/s 2024-09-14T05:13:55.934 INFO:teuthology.orchestra.run.smithi194.stdout: 12 384 33.258 532 KiB/s 2024-09-14T05:13:56.777 INFO:teuthology.orchestra.run.smithi194.stdout: 13 416 34.2416 548 KiB/s 2024-09-14T05:13:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: 14 448 34.4291 551 KiB/s 2024-09-14T05:13:58.845 INFO:teuthology.orchestra.run.smithi194.stdout: 15 480 33.258 532 KiB/s 2024-09-14T05:13:59.640 INFO:teuthology.orchestra.run.smithi194.stdout: 16 512 34.4291 551 KiB/s 2024-09-14T05:14:00.654 INFO:teuthology.orchestra.run.smithi194.stdout: 17 544 33.8986 542 KiB/s 2024-09-14T05:14:01.512 INFO:teuthology.orchestra.run.smithi194.stdout: 18 576 33.7556 540 KiB/s 2024-09-14T05:14:02.535 INFO:teuthology.orchestra.run.smithi194.stdout: 19 608 32.6001 522 KiB/s 2024-09-14T05:14:03.836 INFO:teuthology.orchestra.run.smithi194.stdout: 20 656 35.285 565 KiB/s 2024-09-14T05:14:04.504 INFO:teuthology.orchestra.run.smithi194.stdout: 21 672 32.895 526 KiB/s 2024-09-14T05:14:05.983 INFO:teuthology.orchestra.run.smithi194.stdout: 22 720 33.0333 529 KiB/s 2024-09-14T05:14:06.483 INFO:teuthology.orchestra.run.smithi194.stdout: 23 736 32.2064 515 KiB/s 2024-09-14T05:14:07.965 INFO:teuthology.orchestra.run.smithi194.stdout: 24 784 32.4009 518 KiB/s 2024-09-14T05:14:08.548 INFO:teuthology.orchestra.run.smithi194.stdout: 25 800 30.5605 489 KiB/s 2024-09-14T05:14:09.787 INFO:teuthology.orchestra.run.smithi194.stdout: 26 848 33.3084 533 KiB/s 2024-09-14T05:14:10.847 INFO:teuthology.orchestra.run.smithi194.stdout: 27 880 32.895 526 KiB/s 2024-09-14T05:14:11.791 INFO:teuthology.orchestra.run.smithi194.stdout: 28 912 33.1578 531 KiB/s 2024-09-14T05:14:12.576 INFO:teuthology.orchestra.run.smithi194.stdout: 29 928 31.2232 500 KiB/s 2024-09-14T05:14:13.504 INFO:teuthology.orchestra.run.smithi194.stdout: 30 960 32.2844 517 KiB/s 2024-09-14T05:14:14.474 INFO:teuthology.orchestra.run.smithi194.stdout: 31 992 30.717 491 KiB/s 2024-09-14T05:14:15.484 INFO:teuthology.orchestra.run.smithi194.stdout: 32 1024 31.0348 497 KiB/s 2024-09-14T05:14:16.908 INFO:teuthology.orchestra.run.smithi194.stdout: 33 1072 31.2503 500 KiB/s 2024-09-14T05:14:17.405 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:14:17.405 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:14:17.451 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:14:57.453 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:14:57.453 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:14:57.511 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:14:13 2024","present":true,"empty":false,"clean":false,"allocated_bytes":15925248,"cached_bytes":15925248,"dirty_bytes":3293184,"free_bytes":735694028,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:14:57.513 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:14:57.513 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:14:57.517 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:14:57.517 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 128K --io-total 100G testimage 2024-09-14T05:14:58.029 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:14:59.555 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:14:59.556 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 90.9123 11 MiB/s 2024-09-14T05:15:00.182 INFO:teuthology.orchestra.run.smithi194.stdout: 2 48 55.5575 6.9 MiB/s 2024-09-14T05:15:01.777 INFO:teuthology.orchestra.run.smithi194.stdout: 3 96 40.6991 5.1 MiB/s 2024-09-14T05:15:02.143 INFO:teuthology.orchestra.run.smithi194.stdout: 4 112 41.0797 5.1 MiB/s 2024-09-14T05:15:03.156 INFO:teuthology.orchestra.run.smithi194.stdout: 5 144 38.7235 4.8 MiB/s 2024-09-14T05:15:04.395 INFO:teuthology.orchestra.run.smithi194.stdout: 6 176 29.7285 3.7 MiB/s 2024-09-14T05:15:05.028 INFO:teuthology.orchestra.run.smithi194.stdout: 7 192 29.6795 3.7 MiB/s 2024-09-14T05:15:06.491 INFO:teuthology.orchestra.run.smithi194.stdout: 8 240 30.5353 3.8 MiB/s 2024-09-14T05:15:07.359 INFO:teuthology.orchestra.run.smithi194.stdout: 9 256 27.5871 3.4 MiB/s 2024-09-14T05:15:07.560 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:15:07.560 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:15:07.618 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/2595284434","client":4277,"cookie":139793199567856}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:15:03 2024","present":true,"empty":false,"clean":false,"allocated_bytes":37322752,"cached_bytes":37322752,"dirty_bytes":11010048,"free_bytes":714296524,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:15:07.619 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:15:08.173 INFO:teuthology.orchestra.run.smithi194.stdout: 10 288 28.709 3.6 MiB/s 2024-09-14T05:15:09.448 INFO:teuthology.orchestra.run.smithi194.stdout: 11 320 28.5044 3.6 MiB/s 2024-09-14T05:15:10.469 INFO:teuthology.orchestra.run.smithi194.stdout: 12 352 29.4126 3.7 MiB/s 2024-09-14T05:15:11.036 INFO:teuthology.orchestra.run.smithi194.stdout: 13 368 28.1698 3.5 MiB/s 2024-09-14T05:15:12.732 INFO:teuthology.orchestra.run.smithi194.stdout: 14 416 29.7849 3.7 MiB/s 2024-09-14T05:15:13.250 INFO:teuthology.orchestra.run.smithi194.stdout: 15 432 28.3472 3.5 MiB/s 2024-09-14T05:15:14.420 INFO:teuthology.orchestra.run.smithi194.stdout: 16 464 28.963 3.6 MiB/s 2024-09-14T05:15:15.426 INFO:teuthology.orchestra.run.smithi194.stdout: 17 496 29.0565 3.6 MiB/s 2024-09-14T05:15:16.405 INFO:teuthology.orchestra.run.smithi194.stdout: 18 528 29.807 3.7 MiB/s 2024-09-14T05:15:17.052 INFO:teuthology.orchestra.run.smithi194.stdout: 19 544 29.6304 3.7 MiB/s 2024-09-14T05:15:18.255 INFO:teuthology.orchestra.run.smithi194.stdout: 20 592 31.9752 4.0 MiB/s 2024-09-14T05:15:19.208 INFO:teuthology.orchestra.run.smithi194.stdout: 21 608 30.0759 3.8 MiB/s 2024-09-14T05:15:20.210 INFO:teuthology.orchestra.run.smithi194.stdout: 22 640 30.0759 3.8 MiB/s 2024-09-14T05:15:21.221 INFO:teuthology.orchestra.run.smithi194.stdout: 23 672 29.901 3.7 MiB/s 2024-09-14T05:15:22.183 INFO:teuthology.orchestra.run.smithi194.stdout: 24 704 31.1776 3.9 MiB/s 2024-09-14T05:15:23.344 INFO:teuthology.orchestra.run.smithi194.stdout: 25 736 28.3025 3.5 MiB/s 2024-09-14T05:15:24.030 INFO:teuthology.orchestra.run.smithi194.stdout: 26 752 29.8762 3.7 MiB/s 2024-09-14T05:15:25.066 INFO:teuthology.orchestra.run.smithi194.stdout: 27 784 29.6791 3.7 MiB/s 2024-09-14T05:15:26.112 INFO:teuthology.orchestra.run.smithi194.stdout: 28 816 29.4364 3.7 MiB/s 2024-09-14T05:15:27.129 INFO:teuthology.orchestra.run.smithi194.stdout: 29 848 29.1268 3.6 MiB/s 2024-09-14T05:15:28.316 INFO:teuthology.orchestra.run.smithi194.stdout: 30 880 28.9628 3.6 MiB/s 2024-09-14T05:15:29.733 INFO:teuthology.orchestra.run.smithi194.stdout: 31 912 28.0511 3.5 MiB/s 2024-09-14T05:15:30.321 INFO:teuthology.orchestra.run.smithi194.stdout: 32 928 27.3978 3.4 MiB/s 2024-09-14T05:15:31.409 INFO:teuthology.orchestra.run.smithi194.stdout: 33 960 27.1909 3.4 MiB/s 2024-09-14T05:15:32.598 INFO:teuthology.orchestra.run.smithi194.stdout: 34 992 26.3164 3.3 MiB/s 2024-09-14T05:15:33.110 INFO:teuthology.orchestra.run.smithi194.stdout: 35 1008 26.712 3.3 MiB/s 2024-09-14T05:15:34.374 INFO:teuthology.orchestra.run.smithi194.stdout: 36 1040 27.5873 3.4 MiB/s 2024-09-14T05:15:35.429 INFO:teuthology.orchestra.run.smithi194.stdout: 37 1072 28.1924 3.5 MiB/s 2024-09-14T05:15:36.359 INFO:teuthology.orchestra.run.smithi194.stdout: 38 1104 29.0808 3.6 MiB/s 2024-09-14T05:15:37.617 INFO:teuthology.orchestra.run.smithi194.stdout: 39 1136 28.7098 3.6 MiB/s 2024-09-14T05:15:38.045 INFO:teuthology.orchestra.run.smithi194.stdout: 40 1152 29.1751 3.6 MiB/s 2024-09-14T05:15:39.204 INFO:teuthology.orchestra.run.smithi194.stdout: 41 1184 29.803 3.7 MiB/s 2024-09-14T05:15:40.904 INFO:teuthology.orchestra.run.smithi194.stdout: 42 1232 29.22 3.7 MiB/s 2024-09-14T05:15:41.249 INFO:teuthology.orchestra.run.smithi194.stdout: 43 1248 29.4615 3.7 MiB/s 2024-09-14T05:15:42.646 INFO:teuthology.orchestra.run.smithi194.stdout: 44 1280 28.6411 3.6 MiB/s 2024-09-14T05:15:43.108 INFO:teuthology.orchestra.run.smithi194.stdout: 45 1296 28.4375 3.6 MiB/s 2024-09-14T05:15:44.235 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1328 28.6183 3.6 MiB/s 2024-09-14T05:15:45.364 INFO:teuthology.orchestra.run.smithi194.stdout: 47 1360 28.7267 3.6 MiB/s 2024-09-14T05:15:46.382 INFO:teuthology.orchestra.run.smithi194.stdout: 48 1392 28.0605 3.5 MiB/s 2024-09-14T05:15:46.620 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:15:46.621 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:15:46.644 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:16:26.645 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:16:26.646 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:16:26.703 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:15:43 2024","present":true,"empty":false,"clean":false,"allocated_bytes":189104128,"cached_bytes":189104128,"dirty_bytes":9306112,"free_bytes":562515148,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:16:26.705 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:16:26.705 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:16:26.709 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:16:26.709 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 128K --io-total 100G testimage 2024-09-14T05:16:27.061 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:16:28.514 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:16:28.514 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 78.9488 9.9 MiB/s 2024-09-14T05:16:29.666 INFO:teuthology.orchestra.run.smithi194.stdout: 2 64 45.5589 5.7 MiB/s 2024-09-14T05:16:30.154 INFO:teuthology.orchestra.run.smithi194.stdout: 3 80 42.7815 5.3 MiB/s 2024-09-14T05:16:31.271 INFO:teuthology.orchestra.run.smithi194.stdout: 4 112 38.0506 4.8 MiB/s 2024-09-14T05:16:32.298 INFO:teuthology.orchestra.run.smithi194.stdout: 5 144 36.4637 4.6 MiB/s 2024-09-14T05:16:33.368 INFO:teuthology.orchestra.run.smithi194.stdout: 6 176 29.679 3.7 MiB/s 2024-09-14T05:16:34.717 INFO:teuthology.orchestra.run.smithi194.stdout: 7 208 28.504 3.6 MiB/s 2024-09-14T05:16:35.139 INFO:teuthology.orchestra.run.smithi194.stdout: 8 224 28.8698 3.6 MiB/s 2024-09-14T05:16:36.255 INFO:teuthology.orchestra.run.smithi194.stdout: 9 256 28.9161 3.6 MiB/s 2024-09-14T05:16:36.755 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:16:36.755 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:16:36.813 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/2571992308","client":4293,"cookie":139792528479168}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:16:32 2024","present":true,"empty":false,"clean":false,"allocated_bytes":221216768,"cached_bytes":221216768,"dirty_bytes":10485760,"free_bytes":530402508,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:16:36.814 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:16:37.291 INFO:teuthology.orchestra.run.smithi194.stdout: 10 288 28.8236 3.6 MiB/s 2024-09-14T05:16:38.599 INFO:teuthology.orchestra.run.smithi194.stdout: 11 320 27.5235 3.4 MiB/s 2024-09-14T05:16:39.084 INFO:teuthology.orchestra.run.smithi194.stdout: 12 336 29.3047 3.7 MiB/s 2024-09-14T05:16:40.242 INFO:teuthology.orchestra.run.smithi194.stdout: 13 368 28.236 3.5 MiB/s 2024-09-14T05:16:41.265 INFO:teuthology.orchestra.run.smithi194.stdout: 14 400 28.7318 3.6 MiB/s 2024-09-14T05:16:42.284 INFO:teuthology.orchestra.run.smithi194.stdout: 15 432 28.847 3.6 MiB/s 2024-09-14T05:16:43.526 INFO:teuthology.orchestra.run.smithi194.stdout: 16 464 29.2454 3.7 MiB/s 2024-09-14T05:16:44.389 INFO:teuthology.orchestra.run.smithi194.stdout: 17 496 30.1668 3.8 MiB/s 2024-09-14T05:16:45.088 INFO:teuthology.orchestra.run.smithi194.stdout: 18 512 29.7038 3.7 MiB/s 2024-09-14T05:16:46.105 INFO:teuthology.orchestra.run.smithi194.stdout: 19 544 29.7529 3.7 MiB/s 2024-09-14T05:16:47.142 INFO:teuthology.orchestra.run.smithi194.stdout: 20 576 29.6548 3.7 MiB/s 2024-09-14T05:16:48.425 INFO:teuthology.orchestra.run.smithi194.stdout: 21 608 29.3885 3.7 MiB/s 2024-09-14T05:16:49.653 INFO:teuthology.orchestra.run.smithi194.stdout: 22 640 27.3355 3.4 MiB/s 2024-09-14T05:16:50.229 INFO:teuthology.orchestra.run.smithi194.stdout: 23 656 28.0163 3.5 MiB/s 2024-09-14T05:16:51.374 INFO:teuthology.orchestra.run.smithi194.stdout: 24 688 27.3355 3.4 MiB/s 2024-09-14T05:16:52.352 INFO:teuthology.orchestra.run.smithi194.stdout: 25 720 27.6292 3.5 MiB/s 2024-09-14T05:16:53.599 INFO:teuthology.orchestra.run.smithi194.stdout: 26 752 27.8429 3.5 MiB/s 2024-09-14T05:16:54.140 INFO:teuthology.orchestra.run.smithi194.stdout: 27 768 28.5466 3.6 MiB/s 2024-09-14T05:16:55.285 INFO:teuthology.orchestra.run.smithi194.stdout: 28 800 28.4817 3.6 MiB/s 2024-09-14T05:16:56.421 INFO:teuthology.orchestra.run.smithi194.stdout: 29 832 28.5268 3.6 MiB/s 2024-09-14T05:16:57.365 INFO:teuthology.orchestra.run.smithi194.stdout: 30 864 28.7088 3.6 MiB/s 2024-09-14T05:16:58.503 INFO:teuthology.orchestra.run.smithi194.stdout: 31 896 29.3644 3.7 MiB/s 2024-09-14T05:16:59.335 INFO:teuthology.orchestra.run.smithi194.stdout: 32 912 27.7142 3.5 MiB/s 2024-09-14T05:17:00.211 INFO:teuthology.orchestra.run.smithi194.stdout: 33 944 29.2451 3.7 MiB/s 2024-09-14T05:17:01.460 INFO:teuthology.orchestra.run.smithi194.stdout: 34 976 28.572 3.6 MiB/s 2024-09-14T05:17:02.526 INFO:teuthology.orchestra.run.smithi194.stdout: 35 1008 29.1032 3.6 MiB/s 2024-09-14T05:17:03.687 INFO:teuthology.orchestra.run.smithi194.stdout: 36 1040 27.7783 3.5 MiB/s 2024-09-14T05:17:04.095 INFO:teuthology.orchestra.run.smithi194.stdout: 37 1056 30.2527 3.8 MiB/s 2024-09-14T05:17:05.270 INFO:teuthology.orchestra.run.smithi194.stdout: 38 1088 28.459 3.6 MiB/s 2024-09-14T05:17:06.465 INFO:teuthology.orchestra.run.smithi194.stdout: 39 1120 28.7545 3.6 MiB/s 2024-09-14T05:17:07.425 INFO:teuthology.orchestra.run.smithi194.stdout: 40 1152 28.1916 3.5 MiB/s 2024-09-14T05:17:08.105 INFO:teuthology.orchestra.run.smithi194.stdout: 41 1168 28.9336 3.6 MiB/s 2024-09-14T05:17:09.230 INFO:teuthology.orchestra.run.smithi194.stdout: 42 1200 28.0379 3.5 MiB/s 2024-09-14T05:17:10.586 INFO:teuthology.orchestra.run.smithi194.stdout: 43 1232 27.0885 3.4 MiB/s 2024-09-14T05:17:11.749 INFO:teuthology.orchestra.run.smithi194.stdout: 44 1264 27.2732 3.4 MiB/s 2024-09-14T05:17:12.271 INFO:teuthology.orchestra.run.smithi194.stdout: 45 1280 26.4031 3.3 MiB/s 2024-09-14T05:17:13.216 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1312 28.1915 3.5 MiB/s 2024-09-14T05:17:14.594 INFO:teuthology.orchestra.run.smithi194.stdout: 47 1344 26.8661 3.4 MiB/s 2024-09-14T05:17:15.325 INFO:teuthology.orchestra.run.smithi194.stdout: 48 1360 27.0047 3.4 MiB/s 2024-09-14T05:17:16.582 INFO:teuthology.orchestra.run.smithi194.stdout: 49 1392 26.4686 3.3 MiB/s 2024-09-14T05:17:17.255 INFO:teuthology.orchestra.run.smithi194.stdout: 50 1408 25.6826 3.2 MiB/s 2024-09-14T05:17:18.159 INFO:teuthology.orchestra.run.smithi194.stdout: 51 1424 22.6541 2.8 MiB/s 2024-09-14T05:17:19.459 INFO:teuthology.orchestra.run.smithi194.stdout: 52 1456 23.0077 2.9 MiB/s 2024-09-14T05:17:20.710 INFO:teuthology.orchestra.run.smithi194.stdout: 53 1488 23.7745 3.0 MiB/s 2024-09-14T05:17:21.259 INFO:teuthology.orchestra.run.smithi194.stdout: 54 1504 23.973 3.0 MiB/s 2024-09-14T05:17:22.212 INFO:teuthology.orchestra.run.smithi194.stdout: 55 1520 22.5992 2.8 MiB/s 2024-09-14T05:17:23.599 INFO:teuthology.orchestra.run.smithi194.stdout: 56 1552 23.5298 2.9 MiB/s 2024-09-14T05:17:24.092 INFO:teuthology.orchestra.run.smithi194.stdout: 57 1568 24.18 3.0 MiB/s 2024-09-14T05:17:25.671 INFO:teuthology.orchestra.run.smithi194.stdout: 58 1600 22.5628 2.8 MiB/s 2024-09-14T05:17:26.439 INFO:teuthology.orchestra.run.smithi194.stdout: 59 1616 21.6219 2.7 MiB/s 2024-09-14T05:17:27.108 INFO:teuthology.orchestra.run.smithi194.stdout: 60 1632 22.8575 2.9 MiB/s 2024-09-14T05:17:28.452 INFO:teuthology.orchestra.run.smithi194.stdout: 61 1664 23.0836 2.9 MiB/s 2024-09-14T05:17:29.779 INFO:teuthology.orchestra.run.smithi194.stdout: 62 1696 22.5038 2.8 MiB/s 2024-09-14T05:17:30.306 INFO:teuthology.orchestra.run.smithi194.stdout: 63 1712 24.1799 3.0 MiB/s 2024-09-14T05:17:31.641 INFO:teuthology.orchestra.run.smithi194.stdout: 64 1744 24.5968 3.1 MiB/s 2024-09-14T05:17:32.177 INFO:teuthology.orchestra.run.smithi194.stdout: 65 1760 25.2768 3.2 MiB/s 2024-09-14T05:17:33.557 INFO:teuthology.orchestra.run.smithi194.stdout: 66 1792 25.0787 3.1 MiB/s 2024-09-14T05:17:34.199 INFO:teuthology.orchestra.run.smithi194.stdout: 67 1808 25.3397 3.2 MiB/s 2024-09-14T05:17:35.528 INFO:teuthology.orchestra.run.smithi194.stdout: 68 1840 24.5026 3.1 MiB/s 2024-09-14T05:17:36.156 INFO:teuthology.orchestra.run.smithi194.stdout: 69 1856 24.801 3.1 MiB/s 2024-09-14T05:17:36.814 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:17:36.815 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:17:36.841 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:18:16.842 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:18:16.843 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:18:16.902 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:17:32 2024","present":true,"empty":false,"clean":false,"allocated_bytes":430784512,"cached_bytes":430784512,"dirty_bytes":17563648,"free_bytes":320834764,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:18:16.905 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:18:16.905 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:18:16.909 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:18:16.909 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 128K --io-total 100G testimage 2024-09-14T05:18:18.302 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:18:20.007 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:18:20.007 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 59.1141 7.4 MiB/s 2024-09-14T05:18:20.574 INFO:teuthology.orchestra.run.smithi194.stdout: 2 48 46.5123 5.8 MiB/s 2024-09-14T05:18:21.807 INFO:teuthology.orchestra.run.smithi194.stdout: 3 80 36.754 4.6 MiB/s 2024-09-14T05:18:22.519 INFO:teuthology.orchestra.run.smithi194.stdout: 4 96 33.6948 4.2 MiB/s 2024-09-14T05:18:23.682 INFO:teuthology.orchestra.run.smithi194.stdout: 5 128 32.1146 4.0 MiB/s 2024-09-14T05:18:24.431 INFO:teuthology.orchestra.run.smithi194.stdout: 6 144 25.3168 3.2 MiB/s 2024-09-14T05:18:25.854 INFO:teuthology.orchestra.run.smithi194.stdout: 7 176 24.2428 3.0 MiB/s 2024-09-14T05:18:26.314 INFO:teuthology.orchestra.run.smithi194.stdout: 8 192 24.8671 3.1 MiB/s 2024-09-14T05:18:26.954 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:18:26.954 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:18:27.012 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/1870280846","client":4305,"cookie":139674416878640}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:18:23 2024","present":true,"empty":false,"clean":false,"allocated_bytes":460865536,"cached_bytes":460865536,"dirty_bytes":23592960,"free_bytes":290753740,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:18:27.014 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:18:27.692 INFO:teuthology.orchestra.run.smithi194.stdout: 9 224 24.749 3.1 MiB/s 2024-09-14T05:18:28.428 INFO:teuthology.orchestra.run.smithi194.stdout: 10 240 23.5892 2.9 MiB/s 2024-09-14T05:18:29.312 INFO:teuthology.orchestra.run.smithi194.stdout: 11 256 22.9511 2.9 MiB/s 2024-09-14T05:18:30.370 INFO:teuthology.orchestra.run.smithi194.stdout: 12 288 24.801 3.1 MiB/s 2024-09-14T05:18:31.713 INFO:teuthology.orchestra.run.smithi194.stdout: 13 320 23.704 3.0 MiB/s 2024-09-14T05:18:33.056 INFO:teuthology.orchestra.run.smithi194.stdout: 14 352 23.8631 3.0 MiB/s 2024-09-14T05:18:33.498 INFO:teuthology.orchestra.run.smithi194.stdout: 15 368 25.2568 3.2 MiB/s 2024-09-14T05:18:34.358 INFO:teuthology.orchestra.run.smithi194.stdout: 16 384 25.377 3.2 MiB/s 2024-09-14T05:18:35.479 INFO:teuthology.orchestra.run.smithi194.stdout: 17 416 25.0394 3.1 MiB/s 2024-09-14T05:18:37.038 INFO:teuthology.orchestra.run.smithi194.stdout: 18 448 24.0424 3.0 MiB/s 2024-09-14T05:18:37.650 INFO:teuthology.orchestra.run.smithi194.stdout: 19 464 24.3905 3.0 MiB/s 2024-09-14T05:18:38.335 INFO:teuthology.orchestra.run.smithi194.stdout: 20 480 23.1408 2.9 MiB/s 2024-09-14T05:18:39.422 INFO:teuthology.orchestra.run.smithi194.stdout: 21 512 25.2768 3.2 MiB/s 2024-09-14T05:18:40.390 INFO:teuthology.orchestra.run.smithi194.stdout: 22 528 22.8202 2.9 MiB/s 2024-09-14T05:18:41.670 INFO:teuthology.orchestra.run.smithi194.stdout: 23 560 24.1799 3.0 MiB/s 2024-09-14T05:18:42.895 INFO:teuthology.orchestra.run.smithi194.stdout: 24 592 24.3905 3.0 MiB/s 2024-09-14T05:18:43.590 INFO:teuthology.orchestra.run.smithi194.stdout: 25 608 24.3534 3.0 MiB/s 2024-09-14T05:18:44.403 INFO:teuthology.orchestra.run.smithi194.stdout: 26 624 22.4722 2.8 MiB/s 2024-09-14T05:18:45.411 INFO:teuthology.orchestra.run.smithi194.stdout: 27 656 25.478 3.2 MiB/s 2024-09-14T05:18:46.686 INFO:teuthology.orchestra.run.smithi194.stdout: 28 688 25.4983 3.2 MiB/s 2024-09-14T05:18:47.632 INFO:teuthology.orchestra.run.smithi194.stdout: 29 704 23.6489 3.0 MiB/s 2024-09-14T05:18:48.644 INFO:teuthology.orchestra.run.smithi194.stdout: 30 736 25.3368 3.2 MiB/s 2024-09-14T05:18:49.502 INFO:teuthology.orchestra.run.smithi194.stdout: 31 752 25.118 3.1 MiB/s 2024-09-14T05:18:50.710 INFO:teuthology.orchestra.run.smithi194.stdout: 32 784 24.1695 3.0 MiB/s 2024-09-14T05:18:52.091 INFO:teuthology.orchestra.run.smithi194.stdout: 33 816 23.6867 3.0 MiB/s 2024-09-14T05:18:52.652 INFO:teuthology.orchestra.run.smithi194.stdout: 34 832 25.4986 3.2 MiB/s 2024-09-14T05:18:53.485 INFO:teuthology.orchestra.run.smithi194.stdout: 35 848 23.1412 2.9 MiB/s 2024-09-14T05:18:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: 36 880 23.5648 2.9 MiB/s 2024-09-14T05:18:55.517 INFO:teuthology.orchestra.run.smithi194.stdout: 37 896 23.2953 2.9 MiB/s 2024-09-14T05:18:56.777 INFO:teuthology.orchestra.run.smithi194.stdout: 38 928 23.912 3.0 MiB/s 2024-09-14T05:18:57.976 INFO:teuthology.orchestra.run.smithi194.stdout: 39 960 24.0429 3.0 MiB/s 2024-09-14T05:18:58.945 INFO:teuthology.orchestra.run.smithi194.stdout: 40 976 23.4612 2.9 MiB/s 2024-09-14T05:18:59.517 INFO:teuthology.orchestra.run.smithi194.stdout: 41 992 24.4336 3.1 MiB/s 2024-09-14T05:19:00.657 INFO:teuthology.orchestra.run.smithi194.stdout: 42 1024 24.9035 3.1 MiB/s 2024-09-14T05:19:01.717 INFO:teuthology.orchestra.run.smithi194.stdout: 43 1040 22.6728 2.8 MiB/s 2024-09-14T05:19:02.308 INFO:teuthology.orchestra.run.smithi194.stdout: 44 1056 22.1613 2.8 MiB/s 2024-09-14T05:19:03.725 INFO:teuthology.orchestra.run.smithi194.stdout: 45 1088 23.4121 2.9 MiB/s 2024-09-14T05:19:04.425 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1104 22.8205 2.9 MiB/s 2024-09-14T05:19:05.592 INFO:teuthology.orchestra.run.smithi194.stdout: 47 1136 22.6911 2.8 MiB/s 2024-09-14T05:19:06.499 INFO:teuthology.orchestra.run.smithi194.stdout: 48 1152 23.412 2.9 MiB/s 2024-09-14T05:19:07.858 INFO:teuthology.orchestra.run.smithi194.stdout: 49 1184 23.072 2.9 MiB/s 2024-09-14T05:19:08.482 INFO:teuthology.orchestra.run.smithi194.stdout: 50 1200 23.5498 2.9 MiB/s 2024-09-14T05:19:10.274 INFO:teuthology.orchestra.run.smithi194.stdout: 51 1232 21.8884 2.7 MiB/s 2024-09-14T05:19:10.830 INFO:teuthology.orchestra.run.smithi194.stdout: 52 1248 21.3909 2.7 MiB/s 2024-09-14T05:19:11.348 INFO:teuthology.orchestra.run.smithi194.stdout: 53 1264 23.1029 2.9 MiB/s 2024-09-14T05:19:12.790 INFO:teuthology.orchestra.run.smithi194.stdout: 54 1296 22.7094 2.8 MiB/s 2024-09-14T05:19:13.648 INFO:teuthology.orchestra.run.smithi194.stdout: 55 1312 21.6723 2.7 MiB/s 2024-09-14T05:19:14.315 INFO:teuthology.orchestra.run.smithi194.stdout: 56 1328 23.7394 3.0 MiB/s 2024-09-14T05:19:15.576 INFO:teuthology.orchestra.run.smithi194.stdout: 57 1360 23.6093 3.0 MiB/s 2024-09-14T05:19:17.272 INFO:teuthology.orchestra.run.smithi194.stdout: 58 1392 21.6075 2.7 MiB/s 2024-09-14T05:19:17.922 INFO:teuthology.orchestra.run.smithi194.stdout: 59 1408 21.8073 2.7 MiB/s 2024-09-14T05:19:18.773 INFO:teuthology.orchestra.run.smithi194.stdout: 60 1424 21.8584 2.7 MiB/s 2024-09-14T05:19:19.530 INFO:teuthology.orchestra.run.smithi194.stdout: 61 1440 21.4893 2.7 MiB/s 2024-09-14T05:19:20.707 INFO:teuthology.orchestra.run.smithi194.stdout: 62 1472 21.8243 2.7 MiB/s 2024-09-14T05:19:21.688 INFO:teuthology.orchestra.run.smithi194.stdout: 63 1504 25.3628 3.2 MiB/s 2024-09-14T05:19:22.555 INFO:teuthology.orchestra.run.smithi194.stdout: 64 1552 31.0887 3.9 MiB/s 2024-09-14T05:19:23.446 INFO:teuthology.orchestra.run.smithi194.stdout: 65 1600 37.6397 4.7 MiB/s 2024-09-14T05:19:24.688 INFO:teuthology.orchestra.run.smithi194.stdout: 66 1664 43.4117 5.4 MiB/s 2024-09-14T05:19:27.014 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:19:27.015 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:19:27.025 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:20:07.025 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:20:07.026 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:20:07.084 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:19:23 2024","present":true,"empty":false,"clean":false,"allocated_bytes":591527936,"cached_bytes":591527936,"dirty_bytes":8519680,"free_bytes":160091340,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:20:07.086 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:20:07.086 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:20:07.090 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:20:07.090 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 16K --io-total 100G testimage 2024-09-14T05:20:08.017 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:20:09.122 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:20:09.122 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 166.668 2.6 MiB/s 2024-09-14T05:20:10.122 INFO:teuthology.orchestra.run.smithi194.stdout: 2 48 49.6899 795 KiB/s 2024-09-14T05:20:11.747 INFO:teuthology.orchestra.run.smithi194.stdout: 3 80 32.9221 527 KiB/s 2024-09-14T05:20:12.955 INFO:teuthology.orchestra.run.smithi194.stdout: 4 112 31.0381 497 KiB/s 2024-09-14T05:20:13.830 INFO:teuthology.orchestra.run.smithi194.stdout: 5 128 28.8003 461 KiB/s 2024-09-14T05:20:14.763 INFO:teuthology.orchestra.run.smithi194.stdout: 6 144 19.8443 318 KiB/s 2024-09-14T05:20:16.721 INFO:teuthology.orchestra.run.smithi194.stdout: 8 160 16.9698 272 KiB/s 2024-09-14T05:20:17.137 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:20:17.137 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:20:17.194 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/1806925821","client":4319,"cookie":140061031044080}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:20:13 2024","present":true,"empty":false,"clean":false,"allocated_bytes":483901440,"cached_bytes":483901440,"dirty_bytes":2080768,"free_bytes":267717836,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:20:17.195 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:20:17.239 INFO:teuthology.orchestra.run.smithi194.stdout: 9 176 17.4801 280 KiB/s 2024-09-14T05:20:19.307 INFO:teuthology.orchestra.run.smithi194.stdout: 11 192 12.5946 202 KiB/s 2024-09-14T05:20:20.053 INFO:teuthology.orchestra.run.smithi194.stdout: 12 208 12.8619 206 KiB/s 2024-09-14T05:20:21.261 INFO:teuthology.orchestra.run.smithi194.stdout: 13 288 22.1677 355 KiB/s 2024-09-14T05:20:22.295 INFO:teuthology.orchestra.run.smithi194.stdout: 14 320 28.6947 459 KiB/s 2024-09-14T05:20:23.120 INFO:teuthology.orchestra.run.smithi194.stdout: 15 384 35.3745 566 KiB/s 2024-09-14T05:20:24.136 INFO:teuthology.orchestra.run.smithi194.stdout: 16 464 56.2919 901 KiB/s 2024-09-14T05:20:25.136 INFO:teuthology.orchestra.run.smithi194.stdout: 17 560 69.2374 1.1 MiB/s 2024-09-14T05:20:26.119 INFO:teuthology.orchestra.run.smithi194.stdout: 18 656 75.7208 1.2 MiB/s 2024-09-14T05:20:27.186 INFO:teuthology.orchestra.run.smithi194.stdout: 19 752 88.3082 1.4 MiB/s 2024-09-14T05:20:28.077 INFO:teuthology.orchestra.run.smithi194.stdout: 20 816 87.1678 1.4 MiB/s 2024-09-14T05:20:29.158 INFO:teuthology.orchestra.run.smithi194.stdout: 21 912 89.2438 1.4 MiB/s 2024-09-14T05:20:30.127 INFO:teuthology.orchestra.run.smithi194.stdout: 22 992 86.5392 1.4 MiB/s 2024-09-14T05:20:31.152 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1088 85.7831 1.3 MiB/s 2024-09-14T05:20:32.035 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1168 85.8093 1.3 MiB/s 2024-09-14T05:20:33.193 INFO:teuthology.orchestra.run.smithi194.stdout: 25 1264 87.5008 1.4 MiB/s 2024-09-14T05:20:34.068 INFO:teuthology.orchestra.run.smithi194.stdout: 26 1344 88.0203 1.4 MiB/s 2024-09-14T05:20:35.462 INFO:teuthology.orchestra.run.smithi194.stdout: 27 1440 83.9588 1.3 MiB/s 2024-09-14T05:20:36.031 INFO:teuthology.orchestra.run.smithi194.stdout: 28 1456 75.4724 1.2 MiB/s 2024-09-14T05:20:37.308 INFO:teuthology.orchestra.run.smithi194.stdout: 29 1520 66.7684 1.0 MiB/s 2024-09-14T05:20:38.067 INFO:teuthology.orchestra.run.smithi194.stdout: 30 1552 59.1138 946 KiB/s 2024-09-14T05:20:39.059 INFO:teuthology.orchestra.run.smithi194.stdout: 31 1616 54.4877 872 KiB/s 2024-09-14T05:20:40.209 INFO:teuthology.orchestra.run.smithi194.stdout: 32 1696 53.9179 863 KiB/s 2024-09-14T05:20:41.056 INFO:teuthology.orchestra.run.smithi194.stdout: 33 1776 63.6948 1019 KiB/s 2024-09-14T05:20:42.108 INFO:teuthology.orchestra.run.smithi194.stdout: 34 1856 70.0006 1.1 MiB/s 2024-09-14T05:20:43.066 INFO:teuthology.orchestra.run.smithi194.stdout: 35 1936 76.8007 1.2 MiB/s 2024-09-14T05:20:44.025 INFO:teuthology.orchestra.run.smithi194.stdout: 36 2000 77.3577 1.2 MiB/s 2024-09-14T05:20:45.158 INFO:teuthology.orchestra.run.smithi194.stdout: 37 2112 84.1431 1.3 MiB/s 2024-09-14T05:20:46.357 INFO:teuthology.orchestra.run.smithi194.stdout: 38 2192 78.4913 1.2 MiB/s 2024-09-14T05:20:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: 39 2272 81.3143 1.3 MiB/s 2024-09-14T05:20:48.080 INFO:teuthology.orchestra.run.smithi194.stdout: 40 2352 82.9353 1.3 MiB/s 2024-09-14T05:20:49.142 INFO:teuthology.orchestra.run.smithi194.stdout: 41 2432 84.4417 1.3 MiB/s 2024-09-14T05:20:50.032 INFO:teuthology.orchestra.run.smithi194.stdout: 42 2496 78.7538 1.2 MiB/s 2024-09-14T05:20:51.063 INFO:teuthology.orchestra.run.smithi194.stdout: 43 2592 84.9625 1.3 MiB/s 2024-09-14T05:20:52.175 INFO:teuthology.orchestra.run.smithi194.stdout: 44 2672 80.7762 1.3 MiB/s 2024-09-14T05:20:53.115 INFO:teuthology.orchestra.run.smithi194.stdout: 45 2752 79.492 1.2 MiB/s 2024-09-14T05:20:54.106 INFO:teuthology.orchestra.run.smithi194.stdout: 46 2832 80.516 1.3 MiB/s 2024-09-14T05:20:55.183 INFO:teuthology.orchestra.run.smithi194.stdout: 47 2928 83.8517 1.3 MiB/s 2024-09-14T05:20:56.156 INFO:teuthology.orchestra.run.smithi194.stdout: 48 3024 84.8397 1.3 MiB/s 2024-09-14T05:20:57.081 INFO:teuthology.orchestra.run.smithi194.stdout: 49 3104 88.0921 1.4 MiB/s 2024-09-14T05:20:58.162 INFO:teuthology.orchestra.run.smithi194.stdout: 50 3184 85.5792 1.3 MiB/s 2024-09-14T05:20:59.161 INFO:teuthology.orchestra.run.smithi194.stdout: 51 3264 85.5115 1.3 MiB/s 2024-09-14T05:21:00.065 INFO:teuthology.orchestra.run.smithi194.stdout: 52 3344 85.2467 1.3 MiB/s 2024-09-14T05:21:01.222 INFO:teuthology.orchestra.run.smithi194.stdout: 53 3424 78.9273 1.2 MiB/s 2024-09-14T05:21:02.062 INFO:teuthology.orchestra.run.smithi194.stdout: 54 3504 80.2575 1.3 MiB/s 2024-09-14T05:21:03.138 INFO:teuthology.orchestra.run.smithi194.stdout: 55 3600 83.602 1.3 MiB/s 2024-09-14T05:21:04.141 INFO:teuthology.orchestra.run.smithi194.stdout: 56 3664 80.322 1.3 MiB/s 2024-09-14T05:21:05.068 INFO:teuthology.orchestra.run.smithi194.stdout: 57 3744 79.9368 1.2 MiB/s 2024-09-14T05:21:06.054 INFO:teuthology.orchestra.run.smithi194.stdout: 58 3808 79.4709 1.2 MiB/s 2024-09-14T05:21:07.165 INFO:teuthology.orchestra.run.smithi194.stdout: 59 3872 72.1575 1.1 MiB/s 2024-09-14T05:21:08.079 INFO:teuthology.orchestra.run.smithi194.stdout: 60 3952 71.2557 1.1 MiB/s 2024-09-14T05:21:09.029 INFO:teuthology.orchestra.run.smithi194.stdout: 61 4048 78.4962 1.2 MiB/s 2024-09-14T05:21:10.113 INFO:teuthology.orchestra.run.smithi194.stdout: 62 4128 76.1307 1.2 MiB/s 2024-09-14T05:21:11.062 INFO:teuthology.orchestra.run.smithi194.stdout: 63 4208 79.8729 1.2 MiB/s 2024-09-14T05:21:12.195 INFO:teuthology.orchestra.run.smithi194.stdout: 64 4288 82.606 1.3 MiB/s 2024-09-14T05:21:13.045 INFO:teuthology.orchestra.run.smithi194.stdout: 65 4352 80.5809 1.3 MiB/s 2024-09-14T05:21:14.039 INFO:teuthology.orchestra.run.smithi194.stdout: 66 4432 76.678 1.2 MiB/s 2024-09-14T05:21:14.196 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:21:14.197 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:21:14.207 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:21:54.207 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:21:54.208 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:21:54.265 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:21:13 2024","present":true,"empty":false,"clean":false,"allocated_bytes":373735424,"cached_bytes":373735424,"dirty_bytes":4112384,"free_bytes":377883852,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:21:54.267 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:21:54.267 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:21:54.271 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:21:54.271 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 4K --io-total 100G testimage 2024-09-14T05:21:55.405 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 4096 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:21:57.442 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:21:57.442 INFO:teuthology.orchestra.run.smithi194.stdout: 2 32 375.003 1.5 MiB/s 2024-09-14T05:21:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: 3 128 116.13 465 KiB/s 2024-09-14T05:21:59.459 INFO:teuthology.orchestra.run.smithi194.stdout: 4 224 111.941 448 KiB/s 2024-09-14T05:22:00.512 INFO:teuthology.orchestra.run.smithi194.stdout: 5 304 100.126 401 KiB/s 2024-09-14T05:22:01.454 INFO:teuthology.orchestra.run.smithi194.stdout: 6 384 96.526 386 KiB/s 2024-09-14T05:22:02.539 INFO:teuthology.orchestra.run.smithi194.stdout: 7 480 87.8439 351 KiB/s 2024-09-14T05:22:03.487 INFO:teuthology.orchestra.run.smithi194.stdout: 8 592 94.0041 376 KiB/s 2024-09-14T05:22:04.315 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:22:04.315 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:22:04.374 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/4031237887","client":4337,"cookie":140474891408464}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:22:00 2024","present":true,"empty":false,"clean":false,"allocated_bytes":374235136,"cached_bytes":374235136,"dirty_bytes":401408,"free_bytes":377384140,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:22:04.376 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:22:04.562 INFO:teuthology.orchestra.run.smithi194.stdout: 9 704 93.9712 376 KiB/s 2024-09-14T05:22:05.520 INFO:teuthology.orchestra.run.smithi194.stdout: 10 800 99.0427 396 KiB/s 2024-09-14T05:22:06.627 INFO:teuthology.orchestra.run.smithi194.stdout: 11 896 99.0725 396 KiB/s 2024-09-14T05:22:07.511 INFO:teuthology.orchestra.run.smithi194.stdout: 12 992 102.978 412 KiB/s 2024-09-14T05:22:08.419 INFO:teuthology.orchestra.run.smithi194.stdout: 13 1056 94.081 376 KiB/s 2024-09-14T05:22:09.528 INFO:teuthology.orchestra.run.smithi194.stdout: 14 1152 90.2514 361 KiB/s 2024-09-14T05:22:10.439 INFO:teuthology.orchestra.run.smithi194.stdout: 15 1232 87.8064 351 KiB/s 2024-09-14T05:22:11.524 INFO:teuthology.orchestra.run.smithi194.stdout: 16 1344 91.5049 366 KiB/s 2024-09-14T05:22:12.518 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1456 92.7274 371 KiB/s 2024-09-14T05:22:13.444 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1520 92.3583 369 KiB/s 2024-09-14T05:22:14.409 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1616 95.0836 380 KiB/s 2024-09-14T05:22:15.490 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1744 101.348 405 KiB/s 2024-09-14T05:22:16.442 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1808 94.3106 377 KiB/s 2024-09-14T05:22:17.487 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1888 86.8881 348 KiB/s 2024-09-14T05:22:18.445 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1984 92.8016 371 KiB/s 2024-09-14T05:22:19.484 INFO:teuthology.orchestra.run.smithi194.stdout: 24 2096 94.6389 379 KiB/s 2024-09-14T05:22:20.500 INFO:teuthology.orchestra.run.smithi194.stdout: 25 2192 89.4584 358 KiB/s 2024-09-14T05:22:21.484 INFO:teuthology.orchestra.run.smithi194.stdout: 26 2272 92.0651 368 KiB/s 2024-09-14T05:22:22.418 INFO:teuthology.orchestra.run.smithi194.stdout: 27 2352 94.0811 376 KiB/s 2024-09-14T05:22:23.580 INFO:teuthology.orchestra.run.smithi194.stdout: 28 2464 93.5324 374 KiB/s 2024-09-14T05:22:24.516 INFO:teuthology.orchestra.run.smithi194.stdout: 29 2560 92.2114 369 KiB/s 2024-09-14T05:22:25.458 INFO:teuthology.orchestra.run.smithi194.stdout: 30 2624 87.0983 348 KiB/s 2024-09-14T05:22:26.612 INFO:teuthology.orchestra.run.smithi194.stdout: 31 2720 87.365 349 KiB/s 2024-09-14T05:22:27.542 INFO:teuthology.orchestra.run.smithi194.stdout: 32 2816 90.6265 363 KiB/s 2024-09-14T05:22:28.499 INFO:teuthology.orchestra.run.smithi194.stdout: 33 2928 94.2339 377 KiB/s 2024-09-14T05:22:29.516 INFO:teuthology.orchestra.run.smithi194.stdout: 34 3008 89.6015 358 KiB/s 2024-09-14T05:22:30.498 INFO:teuthology.orchestra.run.smithi194.stdout: 35 3104 95.2397 381 KiB/s 2024-09-14T05:22:31.482 INFO:teuthology.orchestra.run.smithi194.stdout: 36 3200 98.443 394 KiB/s 2024-09-14T05:22:32.574 INFO:teuthology.orchestra.run.smithi194.stdout: 37 3296 95.3153 381 KiB/s 2024-09-14T05:22:33.475 INFO:teuthology.orchestra.run.smithi194.stdout: 38 3360 86.888 348 KiB/s 2024-09-14T05:22:34.506 INFO:teuthology.orchestra.run.smithi194.stdout: 39 3456 89.6732 359 KiB/s 2024-09-14T05:22:35.506 INFO:teuthology.orchestra.run.smithi194.stdout: 40 3568 92.5794 370 KiB/s 2024-09-14T05:22:36.591 INFO:teuthology.orchestra.run.smithi194.stdout: 41 3664 90.8394 363 KiB/s 2024-09-14T05:22:37.560 INFO:teuthology.orchestra.run.smithi194.stdout: 42 3760 93.0995 372 KiB/s 2024-09-14T05:22:38.473 INFO:teuthology.orchestra.run.smithi194.stdout: 43 3872 102.32 409 KiB/s 2024-09-14T05:22:39.431 INFO:teuthology.orchestra.run.smithi194.stdout: 44 3920 94.2339 377 KiB/s 2024-09-14T05:22:40.465 INFO:teuthology.orchestra.run.smithi194.stdout: 45 4016 90.3241 361 KiB/s 2024-09-14T05:22:41.556 INFO:teuthology.orchestra.run.smithi194.stdout: 46 4112 90.2513 361 KiB/s 2024-09-14T05:22:42.505 INFO:teuthology.orchestra.run.smithi194.stdout: 47 4176 84.0758 336 KiB/s 2024-09-14T05:22:43.511 INFO:teuthology.orchestra.run.smithi194.stdout: 48 4272 79.4294 318 KiB/s 2024-09-14T05:22:44.407 INFO:teuthology.orchestra.run.smithi194.stdout: 49 4368 90.0336 360 KiB/s 2024-09-14T05:22:45.659 INFO:teuthology.orchestra.run.smithi194.stdout: 50 4480 89.3697 357 KiB/s 2024-09-14T05:22:46.665 INFO:teuthology.orchestra.run.smithi194.stdout: 51 4576 90.7683 363 KiB/s 2024-09-14T05:22:47.479 INFO:teuthology.orchestra.run.smithi194.stdout: 52 4672 99.7603 399 KiB/s 2024-09-14T05:22:48.479 INFO:teuthology.orchestra.run.smithi194.stdout: 53 4768 99.8406 399 KiB/s 2024-09-14T05:22:49.583 INFO:teuthology.orchestra.run.smithi194.stdout: 54 4848 92.7372 371 KiB/s 2024-09-14T05:22:50.629 INFO:teuthology.orchestra.run.smithi194.stdout: 55 4944 93.3241 373 KiB/s 2024-09-14T05:22:51.470 INFO:teuthology.orchestra.run.smithi194.stdout: 56 5040 96.5878 386 KiB/s 2024-09-14T05:22:52.637 INFO:teuthology.orchestra.run.smithi194.stdout: 57 5136 89.9239 360 KiB/s 2024-09-14T05:22:53.472 INFO:teuthology.orchestra.run.smithi194.stdout: 58 5216 89.7451 359 KiB/s 2024-09-14T05:22:54.459 INFO:teuthology.orchestra.run.smithi194.stdout: 59 5312 95.1615 381 KiB/s 2024-09-14T05:22:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: 60 5424 97.3252 389 KiB/s 2024-09-14T05:22:56.461 INFO:teuthology.orchestra.run.smithi194.stdout: 61 5520 96.1554 385 KiB/s 2024-09-14T05:22:57.376 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:22:57.376 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:22:57.389 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:23:37.390 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:23:37.391 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:23:37.451 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:22:55 2024","present":true,"empty":false,"clean":false,"allocated_bytes":374366208,"cached_bytes":374366208,"dirty_bytes":1155072,"free_bytes":377253068,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:23:37.453 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:23:37.453 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:23:37.457 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:23:37.457 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 16K --io-total 100G testimage 2024-09-14T05:23:38.202 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:23:39.437 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:23:39.437 INFO:teuthology.orchestra.run.smithi194.stdout: 1 80 115.386 1.8 MiB/s 2024-09-14T05:23:40.379 INFO:teuthology.orchestra.run.smithi194.stdout: 2 160 99.3243 1.6 MiB/s 2024-09-14T05:23:41.237 INFO:teuthology.orchestra.run.smithi194.stdout: 3 240 97.2659 1.5 MiB/s 2024-09-14T05:23:42.378 INFO:teuthology.orchestra.run.smithi194.stdout: 4 320 89.0788 1.4 MiB/s 2024-09-14T05:23:43.470 INFO:teuthology.orchestra.run.smithi194.stdout: 5 416 88.8171 1.4 MiB/s 2024-09-14T05:23:44.262 INFO:teuthology.orchestra.run.smithi194.stdout: 6 480 82.92 1.3 MiB/s 2024-09-14T05:23:45.292 INFO:teuthology.orchestra.run.smithi194.stdout: 7 576 84.6229 1.3 MiB/s 2024-09-14T05:23:46.211 INFO:teuthology.orchestra.run.smithi194.stdout: 8 640 80.4517 1.3 MiB/s 2024-09-14T05:23:47.268 INFO:teuthology.orchestra.run.smithi194.stdout: 9 736 85.1077 1.3 MiB/s 2024-09-14T05:23:47.503 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:23:47.503 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:23:47.562 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3684106762","client":4351,"cookie":139895809020912}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:23:43 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375865344,"cached_bytes":375865344,"dirty_bytes":2588672,"free_bytes":375753932,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:23:47.563 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:23:48.435 INFO:teuthology.orchestra.run.smithi194.stdout: 10 832 83.8046 1.3 MiB/s 2024-09-14T05:23:49.382 INFO:teuthology.orchestra.run.smithi194.stdout: 11 912 84.3763 1.3 MiB/s 2024-09-14T05:23:50.210 INFO:teuthology.orchestra.run.smithi194.stdout: 12 976 81.3682 1.3 MiB/s 2024-09-14T05:23:51.402 INFO:teuthology.orchestra.run.smithi194.stdout: 13 1072 83.2062 1.3 MiB/s 2024-09-14T05:23:52.226 INFO:teuthology.orchestra.run.smithi194.stdout: 14 1136 80.6464 1.3 MiB/s 2024-09-14T05:23:53.284 INFO:teuthology.orchestra.run.smithi194.stdout: 15 1232 82.4415 1.3 MiB/s 2024-09-14T05:23:54.268 INFO:teuthology.orchestra.run.smithi194.stdout: 16 1312 81.9013 1.3 MiB/s 2024-09-14T05:23:55.414 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1408 83.0143 1.3 MiB/s 2024-09-14T05:23:56.339 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1488 84.28 1.3 MiB/s 2024-09-14T05:23:57.393 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1568 83.5926 1.3 MiB/s 2024-09-14T05:23:58.293 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1648 83.0683 1.3 MiB/s 2024-09-14T05:23:59.309 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1728 82.4754 1.3 MiB/s 2024-09-14T05:24:00.223 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1824 86.5237 1.4 MiB/s 2024-09-14T05:24:01.275 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1904 84.28 1.3 MiB/s 2024-09-14T05:24:02.325 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1984 84.3484 1.3 MiB/s 2024-09-14T05:24:03.308 INFO:teuthology.orchestra.run.smithi194.stdout: 25 2064 83.002 1.3 MiB/s 2024-09-14T05:24:04.274 INFO:teuthology.orchestra.run.smithi194.stdout: 26 2144 83.8046 1.3 MiB/s 2024-09-14T05:24:05.291 INFO:teuthology.orchestra.run.smithi194.stdout: 27 2224 78.9278 1.2 MiB/s 2024-09-14T05:24:06.265 INFO:teuthology.orchestra.run.smithi194.stdout: 28 2320 83.3346 1.3 MiB/s 2024-09-14T05:24:07.215 INFO:teuthology.orchestra.run.smithi194.stdout: 29 2384 81.8343 1.3 MiB/s 2024-09-14T05:24:08.299 INFO:teuthology.orchestra.run.smithi194.stdout: 30 2480 83.3346 1.3 MiB/s 2024-09-14T05:24:09.365 INFO:teuthology.orchestra.run.smithi194.stdout: 31 2560 81.698 1.3 MiB/s 2024-09-14T05:24:10.383 INFO:teuthology.orchestra.run.smithi194.stdout: 32 2640 81.698 1.3 MiB/s 2024-09-14T05:24:11.348 INFO:teuthology.orchestra.run.smithi194.stdout: 33 2736 81.8265 1.3 MiB/s 2024-09-14T05:24:12.240 INFO:teuthology.orchestra.run.smithi194.stdout: 34 2800 82.7379 1.3 MiB/s 2024-09-14T05:24:13.248 INFO:teuthology.orchestra.run.smithi194.stdout: 35 2896 84.0756 1.3 MiB/s 2024-09-14T05:24:14.206 INFO:teuthology.orchestra.run.smithi194.stdout: 36 2960 82.6458 1.3 MiB/s 2024-09-14T05:24:15.385 INFO:teuthology.orchestra.run.smithi194.stdout: 37 3072 86.3322 1.3 MiB/s 2024-09-14T05:24:16.251 INFO:teuthology.orchestra.run.smithi194.stdout: 38 3136 81.6338 1.3 MiB/s 2024-09-14T05:24:17.331 INFO:teuthology.orchestra.run.smithi194.stdout: 39 3232 84.9069 1.3 MiB/s 2024-09-14T05:24:18.455 INFO:teuthology.orchestra.run.smithi194.stdout: 40 3312 79.8783 1.2 MiB/s 2024-09-14T05:24:19.209 INFO:teuthology.orchestra.run.smithi194.stdout: 41 3392 86.3322 1.3 MiB/s 2024-09-14T05:24:20.308 INFO:teuthology.orchestra.run.smithi194.stdout: 42 3488 84.5541 1.3 MiB/s 2024-09-14T05:24:21.239 INFO:teuthology.orchestra.run.smithi194.stdout: 43 3552 83.4014 1.3 MiB/s 2024-09-14T05:24:21.564 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:24:21.565 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:24:21.574 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:25:01.575 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:25:01.576 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:25:01.635 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:24:18 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375848960,"cached_bytes":375848960,"dirty_bytes":3686400,"free_bytes":375770316,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:25:01.637 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:25:01.637 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:25:01.641 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:25:01.641 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 128K --io-total 100G testimage 2024-09-14T05:25:02.306 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:25:03.482 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:25:03.482 INFO:teuthology.orchestra.run.smithi194.stdout: 1 64 123.458 15 MiB/s 2024-09-14T05:25:04.527 INFO:teuthology.orchestra.run.smithi194.stdout: 2 128 84.9068 11 MiB/s 2024-09-14T05:25:05.312 INFO:teuthology.orchestra.run.smithi194.stdout: 3 176 77.4204 9.7 MiB/s 2024-09-14T05:25:06.349 INFO:teuthology.orchestra.run.smithi194.stdout: 4 240 72.811 9.1 MiB/s 2024-09-14T05:25:07.480 INFO:teuthology.orchestra.run.smithi194.stdout: 5 304 68.8478 8.6 MiB/s 2024-09-14T05:25:08.366 INFO:teuthology.orchestra.run.smithi194.stdout: 6 352 58.9689 7.4 MiB/s 2024-09-14T05:25:09.530 INFO:teuthology.orchestra.run.smithi194.stdout: 7 416 57.6008 7.2 MiB/s 2024-09-14T05:25:10.330 INFO:teuthology.orchestra.run.smithi194.stdout: 8 464 57.3713 7.2 MiB/s 2024-09-14T05:25:11.646 INFO:teuthology.orchestra.run.smithi194.stdout: 9 528 54.3814 6.8 MiB/s 2024-09-14T05:25:11.686 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:25:11.686 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:25:11.746 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/4174748124","client":4365,"cookie":139913592869824}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:25:07 2024","present":true,"empty":false,"clean":false,"allocated_bytes":403423232,"cached_bytes":403423232,"dirty_bytes":15204352,"free_bytes":348196044,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:25:11.748 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:25:12.441 INFO:teuthology.orchestra.run.smithi194.stdout: 10 576 54.8395 6.9 MiB/s 2024-09-14T05:25:13.546 INFO:teuthology.orchestra.run.smithi194.stdout: 11 640 55.5992 6.9 MiB/s 2024-09-14T05:25:14.688 INFO:teuthology.orchestra.run.smithi194.stdout: 12 704 55.858 7.0 MiB/s 2024-09-14T05:25:15.546 INFO:teuthology.orchestra.run.smithi194.stdout: 13 752 55.2578 6.9 MiB/s 2024-09-14T05:25:16.721 INFO:teuthology.orchestra.run.smithi194.stdout: 14 816 56.7384 7.1 MiB/s 2024-09-14T05:25:17.458 INFO:teuthology.orchestra.run.smithi194.stdout: 15 864 57.417 7.2 MiB/s 2024-09-14T05:25:18.529 INFO:teuthology.orchestra.run.smithi194.stdout: 16 928 57.7857 7.2 MiB/s 2024-09-14T05:25:19.495 INFO:teuthology.orchestra.run.smithi194.stdout: 17 976 56.5731 7.1 MiB/s 2024-09-14T05:25:20.437 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1040 58.8724 7.4 MiB/s 2024-09-14T05:25:21.522 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1104 59.9508 7.5 MiB/s 2024-09-14T05:25:22.520 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1152 56.9178 7.1 MiB/s 2024-09-14T05:25:23.345 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1200 56.4792 7.1 MiB/s 2024-09-14T05:25:24.486 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1264 57.6931 7.2 MiB/s 2024-09-14T05:25:25.569 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1328 56.1192 7.0 MiB/s 2024-09-14T05:25:26.544 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1376 54.184 6.8 MiB/s 2024-09-14T05:25:27.328 INFO:teuthology.orchestra.run.smithi194.stdout: 25 1424 56.5261 7.1 MiB/s 2024-09-14T05:25:28.770 INFO:teuthology.orchestra.run.smithi194.stdout: 26 1488 53.0981 6.6 MiB/s 2024-09-14T05:25:29.508 INFO:teuthology.orchestra.run.smithi194.stdout: 27 1536 54.1409 6.8 MiB/s 2024-09-14T05:25:30.344 INFO:teuthology.orchestra.run.smithi194.stdout: 28 1584 53.6021 6.7 MiB/s 2024-09-14T05:25:31.578 INFO:teuthology.orchestra.run.smithi194.stdout: 29 1664 57.189 7.1 MiB/s 2024-09-14T05:25:32.386 INFO:teuthology.orchestra.run.smithi194.stdout: 30 1712 56.9178 7.1 MiB/s 2024-09-14T05:25:33.376 INFO:teuthology.orchestra.run.smithi194.stdout: 31 1776 62.5008 7.8 MiB/s 2024-09-14T05:25:34.501 INFO:teuthology.orchestra.run.smithi194.stdout: 32 1840 60.8982 7.6 MiB/s 2024-09-14T05:25:35.384 INFO:teuthology.orchestra.run.smithi194.stdout: 33 1888 60.3183 7.5 MiB/s 2024-09-14T05:25:36.467 INFO:teuthology.orchestra.run.smithi194.stdout: 34 1952 58.9688 7.4 MiB/s 2024-09-14T05:25:37.435 INFO:teuthology.orchestra.run.smithi194.stdout: 35 2000 57.0983 7.1 MiB/s 2024-09-14T05:25:38.417 INFO:teuthology.orchestra.run.smithi194.stdout: 36 2064 57.1436 7.1 MiB/s 2024-09-14T05:25:39.317 INFO:teuthology.orchestra.run.smithi194.stdout: 37 2112 56.4792 7.1 MiB/s 2024-09-14T05:25:40.633 INFO:teuthology.orchestra.run.smithi194.stdout: 38 2176 54.837 6.9 MiB/s 2024-09-14T05:25:41.725 INFO:teuthology.orchestra.run.smithi194.stdout: 39 2224 51.7117 6.5 MiB/s 2024-09-14T05:25:42.341 INFO:teuthology.orchestra.run.smithi194.stdout: 40 2256 52.1604 6.5 MiB/s 2024-09-14T05:25:43.337 INFO:teuthology.orchestra.run.smithi194.stdout: 41 2304 48.7811 6.1 MiB/s 2024-09-14T05:25:44.349 INFO:teuthology.orchestra.run.smithi194.stdout: 42 2352 47.6954 6.0 MiB/s 2024-09-14T05:25:45.599 INFO:teuthology.orchestra.run.smithi194.stdout: 43 2416 48.3877 6.0 MiB/s 2024-09-14T05:25:46.540 INFO:teuthology.orchestra.run.smithi194.stdout: 44 2464 49.8345 6.2 MiB/s 2024-09-14T05:25:47.540 INFO:teuthology.orchestra.run.smithi194.stdout: 45 2512 49.2314 6.2 MiB/s 2024-09-14T05:25:48.423 INFO:teuthology.orchestra.run.smithi194.stdout: 46 2560 50.3547 6.3 MiB/s 2024-09-14T05:25:49.498 INFO:teuthology.orchestra.run.smithi194.stdout: 47 2624 52.8367 6.6 MiB/s 2024-09-14T05:25:50.390 INFO:teuthology.orchestra.run.smithi194.stdout: 48 2656 50.0841 6.3 MiB/s 2024-09-14T05:25:51.440 INFO:teuthology.orchestra.run.smithi194.stdout: 49 2720 52.2456 6.5 MiB/s 2024-09-14T05:25:52.499 INFO:teuthology.orchestra.run.smithi194.stdout: 50 2768 51.6552 6.5 MiB/s 2024-09-14T05:25:53.423 INFO:teuthology.orchestra.run.smithi194.stdout: 51 2816 51.1597 6.4 MiB/s 2024-09-14T05:25:54.506 INFO:teuthology.orchestra.run.smithi194.stdout: 52 2864 47.8857 6.0 MiB/s 2024-09-14T05:25:55.397 INFO:teuthology.orchestra.run.smithi194.stdout: 53 2912 51.1189 6.4 MiB/s 2024-09-14T05:25:56.589 INFO:teuthology.orchestra.run.smithi194.stdout: 54 2976 49.7287 6.2 MiB/s 2024-09-14T05:25:57.414 INFO:teuthology.orchestra.run.smithi194.stdout: 55 3024 52.0755 6.5 MiB/s 2024-09-14T05:25:58.730 INFO:teuthology.orchestra.run.smithi194.stdout: 56 3088 51.2827 6.4 MiB/s 2024-09-14T05:25:59.863 INFO:teuthology.orchestra.run.smithi194.stdout: 57 3136 50.8228 6.4 MiB/s 2024-09-14T05:26:00.321 INFO:teuthology.orchestra.run.smithi194.stdout: 58 3168 51.9487 6.5 MiB/s 2024-09-14T05:26:01.455 INFO:teuthology.orchestra.run.smithi194.stdout: 59 3232 52.6323 6.6 MiB/s 2024-09-14T05:26:02.696 INFO:teuthology.orchestra.run.smithi194.stdout: 60 3296 51.4768 6.4 MiB/s 2024-09-14T05:26:02.749 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:26:02.749 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:26:02.758 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:26:42.759 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:26:42.760 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:26:42.817 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:26:02 2024","present":true,"empty":false,"clean":false,"allocated_bytes":582778880,"cached_bytes":582778880,"dirty_bytes":22151168,"free_bytes":168840396,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:26:42.819 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:26:42.819 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:26:42.823 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:26:42.823 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 16K --io-total 100G testimage 2024-09-14T05:26:43.168 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:26:44.456 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:26:44.456 INFO:teuthology.orchestra.run.smithi194.stdout: 1 64 104.168 1.6 MiB/s 2024-09-14T05:26:45.281 INFO:teuthology.orchestra.run.smithi194.stdout: 2 112 80.403 1.3 MiB/s 2024-09-14T05:26:46.239 INFO:teuthology.orchestra.run.smithi194.stdout: 3 176 75.3542 1.2 MiB/s 2024-09-14T05:26:48.197 INFO:teuthology.orchestra.run.smithi194.stdout: 5 240 56.7383 908 KiB/s 2024-09-14T05:26:49.846 INFO:teuthology.orchestra.run.smithi194.stdout: 6 256 44.1851 707 KiB/s 2024-09-14T05:26:50.297 INFO:teuthology.orchestra.run.smithi194.stdout: 7 288 38.3304 613 KiB/s 2024-09-14T05:26:51.788 INFO:teuthology.orchestra.run.smithi194.stdout: 8 320 31.9414 511 KiB/s 2024-09-14T05:26:52.613 INFO:teuthology.orchestra.run.smithi194.stdout: 9 352 27.5865 441 KiB/s 2024-09-14T05:26:52.870 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:26:52.870 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:26:52.929 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/544481073","client":4383,"cookie":139842725909504}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:26:48 2024","present":true,"empty":false,"clean":false,"allocated_bytes":579780608,"cached_bytes":579780608,"dirty_bytes":4177920,"free_bytes":171838668,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:26:52.930 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:26:53.279 INFO:teuthology.orchestra.run.smithi194.stdout: 10 368 25.1972 403 KiB/s 2024-09-14T05:26:54.421 INFO:teuthology.orchestra.run.smithi194.stdout: 11 400 31.4414 503 KiB/s 2024-09-14T05:26:57.454 INFO:teuthology.orchestra.run.smithi194.stdout: 14 432 20.1232 322 KiB/s 2024-09-14T05:26:58.258 INFO:teuthology.orchestra.run.smithi194.stdout: 15 464 22.2637 356 KiB/s 2024-09-14T05:27:00.062 INFO:teuthology.orchestra.run.smithi194.stdout: 16 512 21.4825 344 KiB/s 2024-09-14T05:27:00.511 INFO:teuthology.orchestra.run.smithi194.stdout: 17 528 22.1242 354 KiB/s 2024-09-14T05:27:01.338 INFO:teuthology.orchestra.run.smithi194.stdout: 18 592 27.7621 444 KiB/s 2024-09-14T05:27:02.653 INFO:teuthology.orchestra.run.smithi194.stdout: 19 608 33.8726 542 KiB/s 2024-09-14T05:27:03.253 INFO:teuthology.orchestra.run.smithi194.stdout: 20 656 38.462 615 KiB/s 2024-09-14T05:27:05.527 INFO:teuthology.orchestra.run.smithi194.stdout: 22 704 35.1138 562 KiB/s 2024-09-14T05:27:06.977 INFO:teuthology.orchestra.run.smithi194.stdout: 23 768 37.1062 594 KiB/s 2024-09-14T05:27:07.261 INFO:teuthology.orchestra.run.smithi194.stdout: 24 800 35.1356 562 KiB/s 2024-09-14T05:27:08.277 INFO:teuthology.orchestra.run.smithi194.stdout: 25 880 48.3304 773 KiB/s 2024-09-14T05:27:09.285 INFO:teuthology.orchestra.run.smithi194.stdout: 26 928 45.0635 721 KiB/s 2024-09-14T05:27:12.293 INFO:teuthology.orchestra.run.smithi194.stdout: 29 1024 47.3099 757 KiB/s 2024-09-14T05:27:13.229 INFO:teuthology.orchestra.run.smithi194.stdout: 30 1088 51.217 819 KiB/s 2024-09-14T05:27:14.326 INFO:teuthology.orchestra.run.smithi194.stdout: 31 1168 52.0958 834 KiB/s 2024-09-14T05:27:16.142 INFO:teuthology.orchestra.run.smithi194.stdout: 32 1184 38.6773 619 KiB/s 2024-09-14T05:27:16.667 INFO:teuthology.orchestra.run.smithi194.stdout: 33 1200 36.8369 589 KiB/s 2024-09-14T05:27:17.783 INFO:teuthology.orchestra.run.smithi194.stdout: 34 1232 37.8737 606 KiB/s 2024-09-14T05:27:21.442 INFO:teuthology.orchestra.run.smithi194.stdout: 38 1264 21.4323 343 KiB/s 2024-09-14T05:27:22.283 INFO:teuthology.orchestra.run.smithi194.stdout: 39 1296 16.0806 257 KiB/s 2024-09-14T05:27:23.288 INFO:teuthology.orchestra.run.smithi194.stdout: 40 1344 22.3716 358 KiB/s 2024-09-14T05:27:24.182 INFO:teuthology.orchestra.run.smithi194.stdout: 41 1360 21.2995 341 KiB/s 2024-09-14T05:27:26.199 INFO:teuthology.orchestra.run.smithi194.stdout: 43 1376 17.1105 274 KiB/s 2024-09-14T05:27:29.123 INFO:teuthology.orchestra.run.smithi194.stdout: 45 1392 16.6495 266 KiB/s 2024-09-14T05:27:29.373 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1408 15.7926 253 KiB/s 2024-09-14T05:27:30.865 INFO:teuthology.orchestra.run.smithi194.stdout: 47 1424 10.5598 169 KiB/s 2024-09-14T05:27:33.206 INFO:teuthology.orchestra.run.smithi194.stdout: 50 1440 8.86536 142 KiB/s 2024-09-14T05:27:34.265 INFO:teuthology.orchestra.run.smithi194.stdout: 51 1456 9.92075 159 KiB/s 2024-09-14T05:27:35.989 INFO:teuthology.orchestra.run.smithi194.stdout: 52 1472 11.6552 186 KiB/s 2024-09-14T05:27:37.997 INFO:teuthology.orchestra.run.smithi194.stdout: 54 1488 9.27655 148 KiB/s 2024-09-14T05:27:40.121 INFO:teuthology.orchestra.run.smithi194.stdout: 56 1504 8.64315 138 KiB/s 2024-09-14T05:27:40.347 INFO:teuthology.orchestra.run.smithi194.stdout: 57 1520 11.1983 179 KiB/s 2024-09-14T05:27:41.931 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:27:41.931 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:27:41.941 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:28:21.942 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:28:21.943 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:28:22.002 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:27:38 2024","present":true,"empty":false,"clean":false,"allocated_bytes":534425600,"cached_bytes":534425600,"dirty_bytes":3604480,"free_bytes":217193676,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:28:22.004 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:28:22.004 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:28:22.008 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:28:22.008 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 16K --io-total 100G testimage 2024-09-14T05:28:23.249 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:28:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:28:27.947 INFO:teuthology.orchestra.run.smithi194.stdout: 4 32 13.3041 213 KiB/s 2024-09-14T05:28:28.663 INFO:teuthology.orchestra.run.smithi194.stdout: 5 48 14.8014 237 KiB/s 2024-09-14T05:28:29.938 INFO:teuthology.orchestra.run.smithi194.stdout: 6 80 17.1555 274 KiB/s 2024-09-14T05:28:31.405 INFO:teuthology.orchestra.run.smithi194.stdout: 8 96 15.8554 254 KiB/s 2024-09-14T05:28:32.052 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:28:32.053 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:28:32.112 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3854267793","client":4395,"cookie":140451000652816}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:28:28 2024","present":true,"empty":false,"clean":false,"allocated_bytes":532262912,"cached_bytes":532262912,"dirty_bytes":917504,"free_bytes":219356364,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:28:32.113 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:28:32.347 INFO:teuthology.orchestra.run.smithi194.stdout: 9 128 17.9824 288 KiB/s 2024-09-14T05:28:34.154 INFO:teuthology.orchestra.run.smithi194.stdout: 10 208 28.3694 454 KiB/s 2024-09-14T05:28:34.289 INFO:teuthology.orchestra.run.smithi194.stdout: 11 224 31.2951 501 KiB/s 2024-09-14T05:28:35.387 INFO:teuthology.orchestra.run.smithi194.stdout: 12 256 32.2824 517 KiB/s 2024-09-14T05:28:36.687 INFO:teuthology.orchestra.run.smithi194.stdout: 13 336 45.4211 727 KiB/s 2024-09-14T05:28:37.787 INFO:teuthology.orchestra.run.smithi194.stdout: 14 368 44.1186 706 KiB/s 2024-09-14T05:28:38.387 INFO:teuthology.orchestra.run.smithi194.stdout: 15 384 41.5495 665 KiB/s 2024-09-14T05:28:39.262 INFO:teuthology.orchestra.run.smithi194.stdout: 16 432 41.8351 669 KiB/s 2024-09-14T05:28:41.120 INFO:teuthology.orchestra.run.smithi194.stdout: 17 464 36.2883 581 KiB/s 2024-09-14T05:28:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: 18 480 31.0084 496 KiB/s 2024-09-14T05:28:42.319 INFO:teuthology.orchestra.run.smithi194.stdout: 19 544 38.8357 621 KiB/s 2024-09-14T05:28:43.328 INFO:teuthology.orchestra.run.smithi194.stdout: 20 592 42.1061 674 KiB/s 2024-09-14T05:28:45.586 INFO:teuthology.orchestra.run.smithi194.stdout: 22 640 32.8912 526 KiB/s 2024-09-14T05:28:47.269 INFO:teuthology.orchestra.run.smithi194.stdout: 24 656 31.2303 500 KiB/s 2024-09-14T05:28:48.360 INFO:teuthology.orchestra.run.smithi194.stdout: 25 704 31.8732 510 KiB/s 2024-09-14T05:28:49.985 INFO:teuthology.orchestra.run.smithi194.stdout: 26 736 25.0527 401 KiB/s 2024-09-14T05:28:50.485 INFO:teuthology.orchestra.run.smithi194.stdout: 27 752 22.3593 358 KiB/s 2024-09-14T05:28:51.309 INFO:teuthology.orchestra.run.smithi194.stdout: 28 784 25.1577 403 KiB/s 2024-09-14T05:28:53.068 INFO:teuthology.orchestra.run.smithi194.stdout: 29 832 30.3454 486 KiB/s 2024-09-14T05:28:53.376 INFO:teuthology.orchestra.run.smithi194.stdout: 30 864 31.8986 510 KiB/s 2024-09-14T05:28:54.609 INFO:teuthology.orchestra.run.smithi194.stdout: 31 880 31.1425 498 KiB/s 2024-09-14T05:28:56.825 INFO:teuthology.orchestra.run.smithi194.stdout: 33 896 22.7134 363 KiB/s 2024-09-14T05:28:59.358 INFO:teuthology.orchestra.run.smithi194.stdout: 36 928 17.893 286 KiB/s 2024-09-14T05:29:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: 37 944 14.2933 229 KiB/s 2024-09-14T05:29:02.658 INFO:teuthology.orchestra.run.smithi194.stdout: 39 960 10.345 166 KiB/s 2024-09-14T05:29:03.374 INFO:teuthology.orchestra.run.smithi194.stdout: 40 992 12.7798 204 KiB/s 2024-09-14T05:29:05.291 INFO:teuthology.orchestra.run.smithi194.stdout: 42 1024 15.116 242 KiB/s 2024-09-14T05:29:07.092 INFO:teuthology.orchestra.run.smithi194.stdout: 43 1040 14.4781 232 KiB/s 2024-09-14T05:29:07.365 INFO:teuthology.orchestra.run.smithi194.stdout: 44 1072 19.8146 317 KiB/s 2024-09-14T05:29:09.240 INFO:teuthology.orchestra.run.smithi194.stdout: 45 1104 21.8716 350 KiB/s 2024-09-14T05:29:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1120 21.2206 340 KiB/s 2024-09-14T05:29:10.315 INFO:teuthology.orchestra.run.smithi194.stdout: 47 1136 22.2934 357 KiB/s 2024-09-14T05:29:11.114 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:29:11.114 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:29:11.124 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:29:51.126 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:29:51.126 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:29:51.184 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:29:08 2024","present":true,"empty":false,"clean":false,"allocated_bytes":533544960,"cached_bytes":533544960,"dirty_bytes":1490944,"free_bytes":218074316,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:29:51.185 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:29:51.186 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:29:51.189 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:29:51.190 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 16K --io-total 100G testimage 2024-09-14T05:29:52.225 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:29:55.716 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:29:55.716 INFO:teuthology.orchestra.run.smithi194.stdout: 3 32 46.8758 750 KiB/s 2024-09-14T05:29:56.482 INFO:teuthology.orchestra.run.smithi194.stdout: 4 64 44.7435 716 KiB/s 2024-09-14T05:29:57.315 INFO:teuthology.orchestra.run.smithi194.stdout: 5 80 36.586 585 KiB/s 2024-09-14T05:29:59.848 INFO:teuthology.orchestra.run.smithi194.stdout: 7 128 27.9075 447 KiB/s 2024-09-14T05:30:01.235 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:30:01.235 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:30:01.294 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3019347368","client":4409,"cookie":139993653744704}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:29:57 2024","present":true,"empty":false,"clean":false,"allocated_bytes":524582912,"cached_bytes":524582912,"dirty_bytes":1359872,"free_bytes":227036364,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:30:01.296 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:30:02.239 INFO:teuthology.orchestra.run.smithi194.stdout: 10 144 21.198 339 KiB/s 2024-09-14T05:30:03.814 INFO:teuthology.orchestra.run.smithi194.stdout: 11 176 17.7869 285 KiB/s 2024-09-14T05:30:04.830 INFO:teuthology.orchestra.run.smithi194.stdout: 12 192 15.3259 245 KiB/s 2024-09-14T05:30:05.739 INFO:teuthology.orchestra.run.smithi194.stdout: 13 208 15.195 243 KiB/s 2024-09-14T05:30:07.064 INFO:teuthology.orchestra.run.smithi194.stdout: 14 224 13.304 213 KiB/s 2024-09-14T05:30:07.356 INFO:teuthology.orchestra.run.smithi194.stdout: 15 240 18.765 300 KiB/s 2024-09-14T05:30:08.272 INFO:teuthology.orchestra.run.smithi194.stdout: 16 304 28.7001 459 KiB/s 2024-09-14T05:30:09.268 INFO:teuthology.orchestra.run.smithi194.stdout: 17 336 32.4623 519 KiB/s 2024-09-14T05:30:10.588 INFO:teuthology.orchestra.run.smithi194.stdout: 18 384 36.3043 581 KiB/s 2024-09-14T05:30:16.887 INFO:teuthology.orchestra.run.smithi194.stdout: 24 400 17.9229 287 KiB/s 2024-09-14T05:30:17.637 INFO:teuthology.orchestra.run.smithi194.stdout: 25 432 18.6701 299 KiB/s 2024-09-14T05:30:18.387 INFO:teuthology.orchestra.run.smithi194.stdout: 26 480 17.3985 278 KiB/s 2024-09-14T05:30:19.338 INFO:teuthology.orchestra.run.smithi194.stdout: 27 560 22.2491 356 KiB/s 2024-09-14T05:30:20.370 INFO:teuthology.orchestra.run.smithi194.stdout: 28 592 21.2596 340 KiB/s 2024-09-14T05:30:22.411 INFO:teuthology.orchestra.run.smithi194.stdout: 30 608 37.6545 602 KiB/s 2024-09-14T05:30:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: 32 640 29.8856 478 KiB/s 2024-09-14T05:30:26.485 INFO:teuthology.orchestra.run.smithi194.stdout: 34 656 21.7395 348 KiB/s 2024-09-14T05:30:28.918 INFO:teuthology.orchestra.run.smithi194.stdout: 36 672 11.6912 187 KiB/s 2024-09-14T05:30:29.760 INFO:teuthology.orchestra.run.smithi194.stdout: 37 688 10.2216 164 KiB/s 2024-09-14T05:30:30.651 INFO:teuthology.orchestra.run.smithi194.stdout: 38 704 11.6507 186 KiB/s 2024-09-14T05:30:32.976 INFO:teuthology.orchestra.run.smithi194.stdout: 40 720 9.5467 153 KiB/s 2024-09-14T05:30:33.359 INFO:teuthology.orchestra.run.smithi194.stdout: 41 752 13.9618 223 KiB/s 2024-09-14T05:30:34.309 INFO:teuthology.orchestra.run.smithi194.stdout: 42 832 29.6741 475 KiB/s 2024-09-14T05:30:36.425 INFO:teuthology.orchestra.run.smithi194.stdout: 44 864 26.411 423 KiB/s 2024-09-14T05:30:37.250 INFO:teuthology.orchestra.run.smithi194.stdout: 45 928 33.94 543 KiB/s 2024-09-14T05:30:38.233 INFO:teuthology.orchestra.run.smithi194.stdout: 46 1008 54.8372 877 KiB/s 2024-09-14T05:30:38.296 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:30:38.296 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:30:38.306 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:31:18.307 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:31:18.308 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:31:18.365 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:30:37 2024","present":true,"empty":false,"clean":false,"allocated_bytes":374390784,"cached_bytes":374390784,"dirty_bytes":2719744,"free_bytes":377228492,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:31:18.367 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:31:18.367 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:31:18.371 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:31:18.371 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 16K --io-total 100G testimage 2024-09-14T05:31:19.318 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:31:20.409 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:31:20.410 INFO:teuthology.orchestra.run.smithi194.stdout: 1 32 206.902 3.2 MiB/s 2024-09-14T05:31:21.502 INFO:teuthology.orchestra.run.smithi194.stdout: 2 128 108.764 1.7 MiB/s 2024-09-14T05:31:22.442 INFO:teuthology.orchestra.run.smithi194.stdout: 3 208 98.7679 1.5 MiB/s 2024-09-14T05:31:23.459 INFO:teuthology.orchestra.run.smithi194.stdout: 4 288 92.5723 1.4 MiB/s 2024-09-14T05:31:24.465 INFO:teuthology.orchestra.run.smithi194.stdout: 5 384 93.2859 1.5 MiB/s 2024-09-14T05:31:25.408 INFO:teuthology.orchestra.run.smithi194.stdout: 6 464 86.4021 1.4 MiB/s 2024-09-14T05:31:26.551 INFO:teuthology.orchestra.run.smithi194.stdout: 7 560 85.5128 1.3 MiB/s 2024-09-14T05:31:27.475 INFO:teuthology.orchestra.run.smithi194.stdout: 8 624 82.6729 1.3 MiB/s 2024-09-14T05:31:28.416 INFO:teuthology.orchestra.run.smithi194.stdout: 9 688 80.7122 1.3 MiB/s 2024-09-14T05:31:28.416 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:31:28.416 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:31:28.475 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/1911006792","client":4425,"cookie":139790246816576}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:31:24 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375635968,"cached_bytes":375635968,"dirty_bytes":1966080,"free_bytes":375983308,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:31:28.476 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:31:29.341 INFO:teuthology.orchestra.run.smithi194.stdout: 10 768 78.755 1.2 MiB/s 2024-09-14T05:31:30.467 INFO:teuthology.orchestra.run.smithi194.stdout: 11 880 82.2154 1.3 MiB/s 2024-09-14T05:31:31.466 INFO:teuthology.orchestra.run.smithi194.stdout: 12 976 84.6926 1.3 MiB/s 2024-09-14T05:31:32.439 INFO:teuthology.orchestra.run.smithi194.stdout: 13 1056 87.0287 1.4 MiB/s 2024-09-14T05:31:33.370 INFO:teuthology.orchestra.run.smithi194.stdout: 14 1120 87.2396 1.4 MiB/s 2024-09-14T05:31:34.341 INFO:teuthology.orchestra.run.smithi194.stdout: 15 1216 89.6022 1.4 MiB/s 2024-09-14T05:31:35.349 INFO:teuthology.orchestra.run.smithi194.stdout: 16 1312 88.5267 1.4 MiB/s 2024-09-14T05:31:36.449 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1408 86.6795 1.4 MiB/s 2024-09-14T05:31:37.498 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1504 88.6097 1.4 MiB/s 2024-09-14T05:31:38.490 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1600 93.7522 1.5 MiB/s 2024-09-14T05:31:39.390 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1680 91.9198 1.4 MiB/s 2024-09-14T05:31:40.556 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1776 89.0958 1.4 MiB/s 2024-09-14T05:31:41.410 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1856 90.2519 1.4 MiB/s 2024-09-14T05:31:42.414 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1952 91.0591 1.4 MiB/s 2024-09-14T05:31:43.323 INFO:teuthology.orchestra.run.smithi194.stdout: 24 2032 89.3321 1.4 MiB/s 2024-09-14T05:31:44.352 INFO:teuthology.orchestra.run.smithi194.stdout: 25 2112 87.0287 1.4 MiB/s 2024-09-14T05:31:45.414 INFO:teuthology.orchestra.run.smithi194.stdout: 26 2192 85.5987 1.3 MiB/s 2024-09-14T05:31:46.589 INFO:teuthology.orchestra.run.smithi194.stdout: 27 2272 80.3728 1.3 MiB/s 2024-09-14T05:31:47.319 INFO:teuthology.orchestra.run.smithi194.stdout: 28 2336 78.3053 1.2 MiB/s 2024-09-14T05:31:48.438 INFO:teuthology.orchestra.run.smithi194.stdout: 29 2416 75.1191 1.2 MiB/s 2024-09-14T05:31:49.447 INFO:teuthology.orchestra.run.smithi194.stdout: 30 2480 72.2152 1.1 MiB/s 2024-09-14T05:31:50.538 INFO:teuthology.orchestra.run.smithi194.stdout: 31 2576 74.9432 1.2 MiB/s 2024-09-14T05:31:51.504 INFO:teuthology.orchestra.run.smithi194.stdout: 32 2640 74.8594 1.2 MiB/s 2024-09-14T05:31:52.337 INFO:teuthology.orchestra.run.smithi194.stdout: 33 2720 76.4958 1.2 MiB/s 2024-09-14T05:31:53.346 INFO:teuthology.orchestra.run.smithi194.stdout: 34 2816 81.5015 1.3 MiB/s 2024-09-14T05:31:54.386 INFO:teuthology.orchestra.run.smithi194.stdout: 35 2896 84.2807 1.3 MiB/s 2024-09-14T05:31:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: 36 2976 80.0659 1.3 MiB/s 2024-09-14T05:31:56.433 INFO:teuthology.orchestra.run.smithi194.stdout: 37 3056 84.4175 1.3 MiB/s 2024-09-14T05:31:57.444 INFO:teuthology.orchestra.run.smithi194.stdout: 38 3152 84.6415 1.3 MiB/s 2024-09-14T05:31:58.377 INFO:teuthology.orchestra.run.smithi194.stdout: 39 3216 79.4299 1.2 MiB/s 2024-09-14T05:31:59.450 INFO:teuthology.orchestra.run.smithi194.stdout: 40 3312 82.1504 1.3 MiB/s 2024-09-14T05:32:00.594 INFO:teuthology.orchestra.run.smithi194.stdout: 41 3392 82.2804 1.3 MiB/s 2024-09-14T05:32:01.386 INFO:teuthology.orchestra.run.smithi194.stdout: 42 3472 84.0084 1.3 MiB/s 2024-09-14T05:32:02.413 INFO:teuthology.orchestra.run.smithi194.stdout: 43 3568 83.7378 1.3 MiB/s 2024-09-14T05:32:03.394 INFO:teuthology.orchestra.run.smithi194.stdout: 44 3648 86.1951 1.3 MiB/s 2024-09-14T05:32:04.502 INFO:teuthology.orchestra.run.smithi194.stdout: 45 3728 82.3455 1.3 MiB/s 2024-09-14T05:32:05.427 INFO:teuthology.orchestra.run.smithi194.stdout: 46 3808 86.0235 1.3 MiB/s 2024-09-14T05:32:06.352 INFO:teuthology.orchestra.run.smithi194.stdout: 47 3888 83.7378 1.3 MiB/s 2024-09-14T05:32:07.335 INFO:teuthology.orchestra.run.smithi194.stdout: 48 3968 81.2366 1.3 MiB/s 2024-09-14T05:32:08.385 INFO:teuthology.orchestra.run.smithi194.stdout: 49 4064 83.3352 1.3 MiB/s 2024-09-14T05:32:09.443 INFO:teuthology.orchestra.run.smithi194.stdout: 50 4144 84.1443 1.3 MiB/s 2024-09-14T05:32:10.490 INFO:teuthology.orchestra.run.smithi194.stdout: 51 4240 85.3774 1.3 MiB/s 2024-09-14T05:32:11.323 INFO:teuthology.orchestra.run.smithi194.stdout: 52 4304 83.6704 1.3 MiB/s 2024-09-14T05:32:12.434 INFO:teuthology.orchestra.run.smithi194.stdout: 53 4416 87.9141 1.4 MiB/s 2024-09-14T05:32:13.371 INFO:teuthology.orchestra.run.smithi194.stdout: 54 4480 83.469 1.3 MiB/s 2024-09-14T05:32:14.426 INFO:teuthology.orchestra.run.smithi194.stdout: 55 4592 89.8896 1.4 MiB/s 2024-09-14T05:32:15.384 INFO:teuthology.orchestra.run.smithi194.stdout: 56 4672 88.2373 1.4 MiB/s 2024-09-14T05:32:16.443 INFO:teuthology.orchestra.run.smithi194.stdout: 57 4768 90.627 1.4 MiB/s 2024-09-14T05:32:17.508 INFO:teuthology.orchestra.run.smithi194.stdout: 58 4848 85.1083 1.3 MiB/s 2024-09-14T05:32:18.375 INFO:teuthology.orchestra.run.smithi194.stdout: 59 4928 89.4589 1.4 MiB/s 2024-09-14T05:32:19.325 INFO:teuthology.orchestra.run.smithi194.stdout: 60 5008 84.9692 1.3 MiB/s 2024-09-14T05:32:20.416 INFO:teuthology.orchestra.run.smithi194.stdout: 61 5104 85.8525 1.3 MiB/s 2024-09-14T05:32:20.477 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:32:20.478 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:32:20.487 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:33:00.488 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:33:00.489 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:33:00.547 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:32:19 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375783424,"cached_bytes":375783424,"dirty_bytes":2080768,"free_bytes":375835852,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:33:00.549 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:33:00.549 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:33:00.553 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:33:00.553 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 16K --io-total 100G testimage 2024-09-14T05:33:01.372 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 16384 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:33:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:33:02.414 INFO:teuthology.orchestra.run.smithi194.stdout: 1 48 177.783 2.8 MiB/s 2024-09-14T05:33:03.384 INFO:teuthology.orchestra.run.smithi194.stdout: 2 128 108.111 1.7 MiB/s 2024-09-14T05:33:04.502 INFO:teuthology.orchestra.run.smithi194.stdout: 3 224 98.042 1.5 MiB/s 2024-09-14T05:33:05.450 INFO:teuthology.orchestra.run.smithi194.stdout: 4 320 98.9428 1.5 MiB/s 2024-09-14T05:33:06.417 INFO:teuthology.orchestra.run.smithi194.stdout: 5 384 91.6617 1.4 MiB/s 2024-09-14T05:33:07.475 INFO:teuthology.orchestra.run.smithi194.stdout: 6 464 82.1509 1.3 MiB/s 2024-09-14T05:33:08.383 INFO:teuthology.orchestra.run.smithi194.stdout: 7 544 83.2024 1.3 MiB/s 2024-09-14T05:33:09.516 INFO:teuthology.orchestra.run.smithi194.stdout: 8 624 79.6835 1.2 MiB/s 2024-09-14T05:33:10.597 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:33:10.597 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:33:10.657 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/1434599906","client":4437,"cookie":140244305351632}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:33:06 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375783424,"cached_bytes":375783424,"dirty_bytes":2195456,"free_bytes":375835852,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:33:10.658 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:33:10.660 INFO:teuthology.orchestra.run.smithi194.stdout: 9 720 76.7482 1.2 MiB/s 2024-09-14T05:33:11.483 INFO:teuthology.orchestra.run.smithi194.stdout: 10 784 78.9912 1.2 MiB/s 2024-09-14T05:33:12.399 INFO:teuthology.orchestra.run.smithi194.stdout: 11 848 77.9876 1.2 MiB/s 2024-09-14T05:33:13.440 INFO:teuthology.orchestra.run.smithi194.stdout: 12 944 79.0536 1.2 MiB/s 2024-09-14T05:33:14.407 INFO:teuthology.orchestra.run.smithi194.stdout: 13 1024 81.8354 1.3 MiB/s 2024-09-14T05:33:15.476 INFO:teuthology.orchestra.run.smithi194.stdout: 14 1104 79.7365 1.2 MiB/s 2024-09-14T05:33:16.398 INFO:teuthology.orchestra.run.smithi194.stdout: 15 1184 81.3692 1.3 MiB/s 2024-09-14T05:33:17.598 INFO:teuthology.orchestra.run.smithi194.stdout: 16 1264 80.0022 1.3 MiB/s 2024-09-14T05:33:18.399 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1328 77.484 1.2 MiB/s 2024-09-14T05:33:19.391 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1408 77.0487 1.2 MiB/s 2024-09-14T05:33:20.556 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1504 78.6804 1.2 MiB/s 2024-09-14T05:33:21.423 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1568 76.4352 1.2 MiB/s 2024-09-14T05:33:22.495 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1664 81.7684 1.3 MiB/s 2024-09-14T05:33:23.390 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1744 83.4025 1.3 MiB/s 2024-09-14T05:33:24.514 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1824 81.2522 1.3 MiB/s 2024-09-14T05:33:25.472 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1888 78.178 1.2 MiB/s 2024-09-14T05:33:26.468 INFO:teuthology.orchestra.run.smithi194.stdout: 25 1984 82.4111 1.3 MiB/s 2024-09-14T05:33:27.413 INFO:teuthology.orchestra.run.smithi194.stdout: 26 2064 81.303 1.3 MiB/s 2024-09-14T05:33:28.447 INFO:teuthology.orchestra.run.smithi194.stdout: 27 2144 79.0535 1.2 MiB/s 2024-09-14T05:33:29.652 INFO:teuthology.orchestra.run.smithi194.stdout: 28 2240 80.9361 1.3 MiB/s 2024-09-14T05:33:30.489 INFO:teuthology.orchestra.run.smithi194.stdout: 29 2320 86.0581 1.3 MiB/s 2024-09-14T05:33:31.479 INFO:teuthology.orchestra.run.smithi194.stdout: 30 2384 79.8106 1.2 MiB/s 2024-09-14T05:33:32.714 INFO:teuthology.orchestra.run.smithi194.stdout: 31 2480 78.4927 1.2 MiB/s 2024-09-14T05:33:33.453 INFO:teuthology.orchestra.run.smithi194.stdout: 32 2544 79.8744 1.2 MiB/s 2024-09-14T05:33:34.595 INFO:teuthology.orchestra.run.smithi194.stdout: 33 2624 77.672 1.2 MiB/s 2024-09-14T05:33:35.428 INFO:teuthology.orchestra.run.smithi194.stdout: 34 2688 74.4959 1.2 MiB/s 2024-09-14T05:33:36.488 INFO:teuthology.orchestra.run.smithi194.stdout: 35 2784 79.8744 1.2 MiB/s 2024-09-14T05:33:37.637 INFO:teuthology.orchestra.run.smithi194.stdout: 36 2880 81.171 1.3 MiB/s 2024-09-14T05:33:38.477 INFO:teuthology.orchestra.run.smithi194.stdout: 37 2944 79.62 1.2 MiB/s 2024-09-14T05:33:39.461 INFO:teuthology.orchestra.run.smithi194.stdout: 38 3040 85.4583 1.3 MiB/s 2024-09-14T05:33:40.444 INFO:teuthology.orchestra.run.smithi194.stdout: 39 3136 89.3166 1.4 MiB/s 2024-09-14T05:33:41.394 INFO:teuthology.orchestra.run.smithi194.stdout: 40 3216 88.0219 1.4 MiB/s 2024-09-14T05:33:42.577 INFO:teuthology.orchestra.run.smithi194.stdout: 41 3312 87.4517 1.4 MiB/s 2024-09-14T05:33:43.378 INFO:teuthology.orchestra.run.smithi194.stdout: 42 3392 91.431 1.4 MiB/s 2024-09-14T05:33:44.527 INFO:teuthology.orchestra.run.smithi194.stdout: 43 3488 88.4001 1.4 MiB/s 2024-09-14T05:33:45.404 INFO:teuthology.orchestra.run.smithi194.stdout: 44 3568 87.0991 1.4 MiB/s 2024-09-14T05:33:46.410 INFO:teuthology.orchestra.run.smithi194.stdout: 45 3664 89.3165 1.4 MiB/s 2024-09-14T05:33:47.485 INFO:teuthology.orchestra.run.smithi194.stdout: 46 3760 91.2819 1.4 MiB/s 2024-09-14T05:33:47.659 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:33:47.660 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:33:47.669 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:34:27.670 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:34:27.671 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:34:27.733 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:33:46 2024","present":true,"empty":false,"clean":false,"allocated_bytes":376029184,"cached_bytes":376029184,"dirty_bytes":1228800,"free_bytes":375590092,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:34:27.735 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:34:27.735 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:34:27.739 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:34:27.739 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 4K --io-total 100G testimage 2024-09-14T05:34:28.365 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 4096 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:34:30.695 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:34:30.695 INFO:teuthology.orchestra.run.smithi194.stdout: 2 96 92.7175 371 KiB/s 2024-09-14T05:34:31.404 INFO:teuthology.orchestra.run.smithi194.stdout: 3 144 83.3353 333 KiB/s 2024-09-14T05:34:32.440 INFO:teuthology.orchestra.run.smithi194.stdout: 4 224 81.1927 325 KiB/s 2024-09-14T05:34:33.397 INFO:teuthology.orchestra.run.smithi194.stdout: 5 304 81.8015 327 KiB/s 2024-09-14T05:34:34.377 INFO:teuthology.orchestra.run.smithi194.stdout: 6 400 85.0388 340 KiB/s 2024-09-14T05:34:35.419 INFO:teuthology.orchestra.run.smithi194.stdout: 7 480 81.2202 325 KiB/s 2024-09-14T05:34:36.393 INFO:teuthology.orchestra.run.smithi194.stdout: 8 576 86.6099 346 KiB/s 2024-09-14T05:34:37.368 INFO:teuthology.orchestra.run.smithi194.stdout: 9 656 87.6644 351 KiB/s 2024-09-14T05:34:37.785 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:34:37.785 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:34:37.843 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3168896081","client":4451,"cookie":140426640135168}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:34:33 2024","present":true,"empty":false,"clean":false,"allocated_bytes":374517760,"cached_bytes":374517760,"dirty_bytes":430080,"free_bytes":377101516,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:34:37.845 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:34:38.453 INFO:teuthology.orchestra.run.smithi194.stdout: 10 752 88.6097 354 KiB/s 2024-09-14T05:34:39.435 INFO:teuthology.orchestra.run.smithi194.stdout: 11 848 88.6097 354 KiB/s 2024-09-14T05:34:40.484 INFO:teuthology.orchestra.run.smithi194.stdout: 12 960 94.789 379 KiB/s 2024-09-14T05:34:41.444 INFO:teuthology.orchestra.run.smithi194.stdout: 13 1056 95.0141 380 KiB/s 2024-09-14T05:34:42.576 INFO:teuthology.orchestra.run.smithi194.stdout: 14 1152 95.2403 381 KiB/s 2024-09-14T05:34:43.463 INFO:teuthology.orchestra.run.smithi194.stdout: 15 1248 98.9648 396 KiB/s 2024-09-14T05:34:44.375 INFO:teuthology.orchestra.run.smithi194.stdout: 16 1344 100.326 401 KiB/s 2024-09-14T05:34:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1440 96.1561 385 KiB/s 2024-09-14T05:34:46.467 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1536 95.6198 382 KiB/s 2024-09-14T05:34:47.487 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1632 97.7221 391 KiB/s 2024-09-14T05:34:48.465 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1744 99.2023 397 KiB/s 2024-09-14T05:34:49.392 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1840 98.8859 396 KiB/s 2024-09-14T05:34:50.383 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1920 97.8018 391 KiB/s 2024-09-14T05:34:51.374 INFO:teuthology.orchestra.run.smithi194.stdout: 23 2032 101.062 404 KiB/s 2024-09-14T05:34:52.482 INFO:teuthology.orchestra.run.smithi194.stdout: 24 2144 102.566 410 KiB/s 2024-09-14T05:34:53.522 INFO:teuthology.orchestra.run.smithi194.stdout: 25 2224 95.0141 380 KiB/s 2024-09-14T05:34:54.499 INFO:teuthology.orchestra.run.smithi194.stdout: 26 2320 94.046 376 KiB/s 2024-09-14T05:34:55.432 INFO:teuthology.orchestra.run.smithi194.stdout: 27 2416 98.259 393 KiB/s 2024-09-14T05:34:56.498 INFO:teuthology.orchestra.run.smithi194.stdout: 28 2512 93.679 375 KiB/s 2024-09-14T05:34:57.382 INFO:teuthology.orchestra.run.smithi194.stdout: 29 2592 91.4307 366 KiB/s 2024-09-14T05:34:58.416 INFO:teuthology.orchestra.run.smithi194.stdout: 30 2688 94.696 379 KiB/s 2024-09-14T05:34:59.370 INFO:teuthology.orchestra.run.smithi194.stdout: 31 2784 95.2403 381 KiB/s 2024-09-14T05:35:00.381 INFO:teuthology.orchestra.run.smithi194.stdout: 32 2896 97.0111 388 KiB/s 2024-09-14T05:35:01.439 INFO:teuthology.orchestra.run.smithi194.stdout: 33 2992 97.0896 388 KiB/s 2024-09-14T05:35:02.443 INFO:teuthology.orchestra.run.smithi194.stdout: 34 3072 94.8638 379 KiB/s 2024-09-14T05:35:03.506 INFO:teuthology.orchestra.run.smithi194.stdout: 35 3168 94.3418 377 KiB/s 2024-09-14T05:35:04.514 INFO:teuthology.orchestra.run.smithi194.stdout: 36 3280 96.4252 386 KiB/s 2024-09-14T05:35:05.415 INFO:teuthology.orchestra.run.smithi194.stdout: 37 3360 92.2119 369 KiB/s 2024-09-14T05:35:06.383 INFO:teuthology.orchestra.run.smithi194.stdout: 38 3440 90.6903 363 KiB/s 2024-09-14T05:35:07.441 INFO:teuthology.orchestra.run.smithi194.stdout: 39 3536 92.8021 371 KiB/s 2024-09-14T05:35:08.387 INFO:teuthology.orchestra.run.smithi194.stdout: 40 3632 95.0841 380 KiB/s 2024-09-14T05:35:09.538 INFO:teuthology.orchestra.run.smithi194.stdout: 41 3728 89.103 356 KiB/s 2024-09-14T05:35:10.554 INFO:teuthology.orchestra.run.smithi194.stdout: 42 3824 90.2042 361 KiB/s 2024-09-14T05:35:11.377 INFO:teuthology.orchestra.run.smithi194.stdout: 43 3920 96.079 384 KiB/s 2024-09-14T05:35:12.529 INFO:teuthology.orchestra.run.smithi194.stdout: 44 4032 97.4864 390 KiB/s 2024-09-14T05:35:13.533 INFO:teuthology.orchestra.run.smithi194.stdout: 45 4128 96.3502 385 KiB/s 2024-09-14T05:35:14.387 INFO:teuthology.orchestra.run.smithi194.stdout: 46 4208 99.0939 396 KiB/s 2024-09-14T05:35:15.470 INFO:teuthology.orchestra.run.smithi194.stdout: 47 4304 97.722 391 KiB/s 2024-09-14T05:35:16.495 INFO:teuthology.orchestra.run.smithi194.stdout: 48 4416 96.9529 388 KiB/s 2024-09-14T05:35:17.506 INFO:teuthology.orchestra.run.smithi194.stdout: 49 4496 93.1747 373 KiB/s 2024-09-14T05:35:18.467 INFO:teuthology.orchestra.run.smithi194.stdout: 50 4592 94.0816 376 KiB/s 2024-09-14T05:35:19.552 INFO:teuthology.orchestra.run.smithi194.stdout: 51 4704 95.9773 384 KiB/s 2024-09-14T05:35:20.461 INFO:teuthology.orchestra.run.smithi194.stdout: 52 4800 99.2816 397 KiB/s 2024-09-14T05:35:21.457 INFO:teuthology.orchestra.run.smithi194.stdout: 53 4896 96.6983 387 KiB/s 2024-09-14T05:35:22.427 INFO:teuthology.orchestra.run.smithi194.stdout: 54 5008 104.067 416 KiB/s 2024-09-14T05:35:23.370 INFO:teuthology.orchestra.run.smithi194.stdout: 55 5088 101.062 404 KiB/s 2024-09-14T05:35:24.669 INFO:teuthology.orchestra.run.smithi194.stdout: 56 5200 96.9529 388 KiB/s 2024-09-14T05:35:25.468 INFO:teuthology.orchestra.run.smithi194.stdout: 57 5280 95.9253 384 KiB/s 2024-09-14T05:35:26.435 INFO:teuthology.orchestra.run.smithi194.stdout: 58 5392 99.6806 399 KiB/s 2024-09-14T05:35:27.368 INFO:teuthology.orchestra.run.smithi194.stdout: 59 5488 97.1681 389 KiB/s 2024-09-14T05:35:28.404 INFO:teuthology.orchestra.run.smithi194.stdout: 60 5568 95.3916 382 KiB/s 2024-09-14T05:35:29.572 INFO:teuthology.orchestra.run.smithi194.stdout: 61 5680 97.8814 392 KiB/s 2024-09-14T05:35:30.419 INFO:teuthology.orchestra.run.smithi194.stdout: 62 5776 100.164 401 KiB/s 2024-09-14T05:35:31.376 INFO:teuthology.orchestra.run.smithi194.stdout: 63 5872 97.0895 388 KiB/s 2024-09-14T05:35:32.453 INFO:teuthology.orchestra.run.smithi194.stdout: 64 5984 97.5631 390 KiB/s 2024-09-14T05:35:33.525 INFO:teuthology.orchestra.run.smithi194.stdout: 65 6064 96.8771 388 KiB/s 2024-09-14T05:35:33.846 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:35:33.846 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:35:33.856 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:36:13.856 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:36:13.857 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:36:13.916 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:35:33 2024","present":true,"empty":false,"clean":false,"allocated_bytes":375943168,"cached_bytes":375943168,"dirty_bytes":626688,"free_bytes":375676108,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:36:13.918 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:36:13.919 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:36:13.923 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:36:13.923 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 128K --io-total 100G testimage 2024-09-14T05:36:14.262 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:36:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:36:15.392 INFO:teuthology.orchestra.run.smithi194.stdout: 1 64 116.281 15 MiB/s 2024-09-14T05:36:16.327 INFO:teuthology.orchestra.run.smithi194.stdout: 2 112 79.0139 9.9 MiB/s 2024-09-14T05:36:17.468 INFO:teuthology.orchestra.run.smithi194.stdout: 3 176 69.4659 8.7 MiB/s 2024-09-14T05:36:18.285 INFO:teuthology.orchestra.run.smithi194.stdout: 4 224 67.0404 8.4 MiB/s 2024-09-14T05:36:19.356 INFO:teuthology.orchestra.run.smithi194.stdout: 5 288 65.2934 8.2 MiB/s 2024-09-14T05:36:20.567 INFO:teuthology.orchestra.run.smithi194.stdout: 6 352 55.6425 7.0 MiB/s 2024-09-14T05:36:21.634 INFO:teuthology.orchestra.run.smithi194.stdout: 7 416 57.2301 7.2 MiB/s 2024-09-14T05:36:22.576 INFO:teuthology.orchestra.run.smithi194.stdout: 8 464 56.3833 7.0 MiB/s 2024-09-14T05:36:23.350 INFO:teuthology.orchestra.run.smithi194.stdout: 9 512 56.8283 7.1 MiB/s 2024-09-14T05:36:23.969 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:36:23.970 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:36:24.029 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/1112762793","client":4463,"cookie":139922585457776}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:36:19 2024","present":true,"empty":false,"clean":false,"allocated_bytes":402960384,"cached_bytes":402960384,"dirty_bytes":10878976,"free_bytes":348658892,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:36:24.030 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:36:24.450 INFO:teuthology.orchestra.run.smithi194.stdout: 10 576 56.5604 7.1 MiB/s 2024-09-14T05:36:25.433 INFO:teuthology.orchestra.run.smithi194.stdout: 11 640 59.2117 7.4 MiB/s 2024-09-14T05:36:26.325 INFO:teuthology.orchestra.run.smithi194.stdout: 12 688 58.0216 7.3 MiB/s 2024-09-14T05:36:27.275 INFO:teuthology.orchestra.run.smithi194.stdout: 13 752 61.33 7.7 MiB/s 2024-09-14T05:36:28.450 INFO:teuthology.orchestra.run.smithi194.stdout: 14 816 59.6558 7.5 MiB/s 2024-09-14T05:36:29.367 INFO:teuthology.orchestra.run.smithi194.stdout: 15 880 61.8401 7.7 MiB/s 2024-09-14T05:36:30.308 INFO:teuthology.orchestra.run.smithi194.stdout: 16 944 62.3474 7.8 MiB/s 2024-09-14T05:36:31.395 INFO:teuthology.orchestra.run.smithi194.stdout: 17 1024 66.2474 8.3 MiB/s 2024-09-14T05:36:32.507 INFO:teuthology.orchestra.run.smithi194.stdout: 18 1088 64.1726 8.0 MiB/s 2024-09-14T05:36:33.290 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1136 66.1174 8.3 MiB/s 2024-09-14T05:36:34.298 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1200 64.9369 8.1 MiB/s 2024-09-14T05:36:35.416 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1264 62.5997 7.8 MiB/s 2024-09-14T05:36:36.298 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1328 61.9922 7.7 MiB/s 2024-09-14T05:36:37.476 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1376 57.9262 7.2 MiB/s 2024-09-14T05:36:38.292 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1424 57.5098 7.2 MiB/s 2024-09-14T05:36:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: 25 1488 57.3723 7.2 MiB/s 2024-09-14T05:36:40.521 INFO:teuthology.orchestra.run.smithi194.stdout: 26 1552 56.4281 7.1 MiB/s 2024-09-14T05:36:41.348 INFO:teuthology.orchestra.run.smithi194.stdout: 27 1600 53.8418 6.7 MiB/s 2024-09-14T05:36:42.330 INFO:teuthology.orchestra.run.smithi194.stdout: 28 1664 59.3588 7.4 MiB/s 2024-09-14T05:36:43.031 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:36:43.031 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:36:43.041 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:37:23.042 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:37:23.043 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:37:23.101 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:36:39 2024","present":true,"empty":false,"clean":false,"allocated_bytes":506245120,"cached_bytes":506245120,"dirty_bytes":12189696,"free_bytes":245374156,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:37:23.103 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:37:23.104 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:37:23.111 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:37:23.111 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 4K --io-total 100G testimage 2024-09-14T05:37:24.116 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 4096 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:37:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:37:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: 1 64 114.946 460 KiB/s 2024-09-14T05:37:26.190 INFO:teuthology.orchestra.run.smithi194.stdout: 2 144 92.1685 369 KiB/s 2024-09-14T05:37:28.548 INFO:teuthology.orchestra.run.smithi194.stdout: 4 176 46.8763 188 KiB/s 2024-09-14T05:37:29.274 INFO:teuthology.orchestra.run.smithi194.stdout: 5 192 43.1548 173 KiB/s 2024-09-14T05:37:30.931 INFO:teuthology.orchestra.run.smithi194.stdout: 6 208 34.5902 138 KiB/s 2024-09-14T05:37:33.155 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:37:33.156 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:37:33.215 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/2588195778","client":4475,"cookie":140098544899056}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:37:29 2024","present":true,"empty":false,"clean":false,"allocated_bytes":478212096,"cached_bytes":478212096,"dirty_bytes":802816,"free_bytes":273407180,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:37:33.217 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:37:33.231 INFO:teuthology.orchestra.run.smithi194.stdout: 9 224 19.8025 79 KiB/s 2024-09-14T05:37:35.197 INFO:teuthology.orchestra.run.smithi194.stdout: 11 336 21.315 85 KiB/s 2024-09-14T05:37:37.188 INFO:teuthology.orchestra.run.smithi194.stdout: 13 352 20.3709 81 KiB/s 2024-09-14T05:37:39.329 INFO:teuthology.orchestra.run.smithi194.stdout: 15 368 17.5025 70 KiB/s 2024-09-14T05:37:42.212 INFO:teuthology.orchestra.run.smithi194.stdout: 18 400 17.0157 68 KiB/s 2024-09-14T05:37:43.862 INFO:teuthology.orchestra.run.smithi194.stdout: 19 416 18.0524 72 KiB/s 2024-09-14T05:37:45.370 INFO:teuthology.orchestra.run.smithi194.stdout: 21 432 9.43796 38 KiB/s 2024-09-14T05:37:46.199 INFO:teuthology.orchestra.run.smithi194.stdout: 22 512 17.7626 71 KiB/s 2024-09-14T05:37:47.224 INFO:teuthology.orchestra.run.smithi194.stdout: 23 608 30.3961 122 KiB/s 2024-09-14T05:37:48.218 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:37:48.219 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:37:48.229 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:38:28.230 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:38:28.231 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:38:28.290 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:37:44 2024","present":true,"empty":false,"clean":false,"allocated_bytes":430821376,"cached_bytes":430821376,"dirty_bytes":757760,"free_bytes":320797900,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:38:28.291 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:38:28.291 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:38:28.295 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:38:28.295 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 128K --io-total 100G testimage 2024-09-14T05:38:29.356 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 131072 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:38:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:38:31.579 INFO:teuthology.orchestra.run.smithi194.stdout: 2 48 76.9253 9.6 MiB/s 2024-09-14T05:38:32.462 INFO:teuthology.orchestra.run.smithi194.stdout: 3 96 65.27 8.2 MiB/s 2024-09-14T05:38:33.370 INFO:teuthology.orchestra.run.smithi194.stdout: 4 144 60.8846 7.6 MiB/s 2024-09-14T05:38:34.620 INFO:teuthology.orchestra.run.smithi194.stdout: 5 224 61.9853 7.7 MiB/s 2024-09-14T05:38:35.570 INFO:teuthology.orchestra.run.smithi194.stdout: 6 272 59.7032 7.5 MiB/s 2024-09-14T05:38:36.403 INFO:teuthology.orchestra.run.smithi194.stdout: 7 320 56.3864 7.0 MiB/s 2024-09-14T05:38:37.470 INFO:teuthology.orchestra.run.smithi194.stdout: 8 384 57.4637 7.2 MiB/s 2024-09-14T05:38:38.338 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:38:38.338 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:38:38.397 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/3253393924","client":4489,"cookie":140635550028832}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:38:34 2024","present":true,"empty":false,"clean":false,"allocated_bytes":453767168,"cached_bytes":453767168,"dirty_bytes":11796480,"free_bytes":297852108,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:38:38.398 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:38:38.587 INFO:teuthology.orchestra.run.smithi194.stdout: 9 448 58.3286 7.3 MiB/s 2024-09-14T05:38:39.380 INFO:teuthology.orchestra.run.smithi194.stdout: 10 496 57.1445 7.1 MiB/s 2024-09-14T05:38:40.544 INFO:teuthology.orchestra.run.smithi194.stdout: 11 560 57.926 7.2 MiB/s 2024-09-14T05:38:41.502 INFO:teuthology.orchestra.run.smithi194.stdout: 12 608 56.4279 7.1 MiB/s 2024-09-14T05:38:42.427 INFO:teuthology.orchestra.run.smithi194.stdout: 13 672 58.16 7.3 MiB/s 2024-09-14T05:38:43.543 INFO:teuthology.orchestra.run.smithi194.stdout: 14 736 58.0662 7.3 MiB/s 2024-09-14T05:38:44.529 INFO:teuthology.orchestra.run.smithi194.stdout: 15 800 59.0079 7.4 MiB/s 2024-09-14T05:38:45.502 INFO:teuthology.orchestra.run.smithi194.stdout: 16 848 58.0194 7.3 MiB/s 2024-09-14T05:38:46.435 INFO:teuthology.orchestra.run.smithi194.stdout: 17 912 61.69 7.7 MiB/s 2024-09-14T05:38:47.385 INFO:teuthology.orchestra.run.smithi194.stdout: 18 976 61.292 7.7 MiB/s 2024-09-14T05:38:48.602 INFO:teuthology.orchestra.run.smithi194.stdout: 19 1056 63.2929 7.9 MiB/s 2024-09-14T05:38:49.509 INFO:teuthology.orchestra.run.smithi194.stdout: 20 1104 60.9969 7.6 MiB/s 2024-09-14T05:38:50.517 INFO:teuthology.orchestra.run.smithi194.stdout: 21 1152 60.6078 7.6 MiB/s 2024-09-14T05:38:51.376 INFO:teuthology.orchestra.run.smithi194.stdout: 22 1200 58.2541 7.3 MiB/s 2024-09-14T05:38:52.792 INFO:teuthology.orchestra.run.smithi194.stdout: 23 1264 53.2559 6.7 MiB/s 2024-09-14T05:38:53.371 INFO:teuthology.orchestra.run.smithi194.stdout: 24 1296 50.2948 6.3 MiB/s 2024-09-14T05:38:54.561 INFO:teuthology.orchestra.run.smithi194.stdout: 25 1360 50.6744 6.3 MiB/s 2024-09-14T05:38:55.375 INFO:teuthology.orchestra.run.smithi194.stdout: 26 1408 52.7197 6.6 MiB/s 2024-09-14T05:38:56.358 INFO:teuthology.orchestra.run.smithi194.stdout: 27 1456 51.3658 6.4 MiB/s 2024-09-14T05:38:57.825 INFO:teuthology.orchestra.run.smithi194.stdout: 28 1520 50.8354 6.4 MiB/s 2024-09-14T05:38:58.541 INFO:teuthology.orchestra.run.smithi194.stdout: 29 1536 46.405 5.8 MiB/s 2024-09-14T05:38:59.385 INFO:teuthology.orchestra.run.smithi194.stdout: 30 1552 39.8021 5.0 MiB/s 2024-09-14T05:39:00.624 INFO:teuthology.orchestra.run.smithi194.stdout: 31 1584 33.5375 4.2 MiB/s 2024-09-14T05:39:01.885 INFO:teuthology.orchestra.run.smithi194.stdout: 32 1616 28.9444 3.6 MiB/s 2024-09-14T05:39:02.399 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:39:02.399 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:39:02.412 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:39:42.413 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:39:42.414 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:39:42.475 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:38:59 2024","present":true,"empty":false,"clean":false,"allocated_bytes":616919040,"cached_bytes":616919040,"dirty_bytes":12713984,"free_bytes":134700236,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:39:42.476 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:39:42.477 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:39:42.481 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:39:42.481 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 4K --io-total 100G testimage 2024-09-14T05:39:43.567 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 4096 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:39:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:39:44.725 INFO:teuthology.orchestra.run.smithi194.stdout: 1 48 173.917 696 KiB/s 2024-09-14T05:39:45.833 INFO:teuthology.orchestra.run.smithi194.stdout: 2 96 75.8827 304 KiB/s 2024-09-14T05:39:47.008 INFO:teuthology.orchestra.run.smithi194.stdout: 3 128 54.382 218 KiB/s 2024-09-14T05:39:48.366 INFO:teuthology.orchestra.run.smithi194.stdout: 4 160 43.9133 176 KiB/s 2024-09-14T05:39:49.408 INFO:teuthology.orchestra.run.smithi194.stdout: 5 176 38.0057 152 KiB/s 2024-09-14T05:39:50.674 INFO:teuthology.orchestra.run.smithi194.stdout: 7 192 24.2104 97 KiB/s 2024-09-14T05:39:52.015 INFO:teuthology.orchestra.run.smithi194.stdout: 8 224 20.6991 83 KiB/s 2024-09-14T05:39:52.527 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:39:52.527 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:39:52.585 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/77466580","client":4501,"cookie":140477106000784}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:39:48 2024","present":true,"empty":false,"clean":false,"allocated_bytes":611553280,"cached_bytes":611553280,"dirty_bytes":745472,"free_bytes":140065996,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:39:52.587 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:39:54.698 INFO:teuthology.orchestra.run.smithi194.stdout: 11 240 14.5609 58 KiB/s 2024-09-14T05:39:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: 12 320 22.1735 89 KiB/s 2024-09-14T05:39:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: 14 352 19.9551 80 KiB/s 2024-09-14T05:40:00.031 INFO:teuthology.orchestra.run.smithi194.stdout: 16 368 18.8119 75 KiB/s 2024-09-14T05:40:00.814 INFO:teuthology.orchestra.run.smithi194.stdout: 17 384 18.1905 73 KiB/s 2024-09-14T05:40:01.830 INFO:teuthology.orchestra.run.smithi194.stdout: 18 400 22.4346 90 KiB/s 2024-09-14T05:40:03.480 INFO:teuthology.orchestra.run.smithi194.stdout: 19 416 12.1584 49 KiB/s 2024-09-14T05:40:06.055 INFO:teuthology.orchestra.run.smithi194.stdout: 22 432 10.2252 41 KiB/s 2024-09-14T05:40:07.037 INFO:teuthology.orchestra.run.smithi194.stdout: 23 448 11.4288 46 KiB/s 2024-09-14T05:40:09.404 INFO:teuthology.orchestra.run.smithi194.stdout: 25 464 9.31121 37 KiB/s 2024-09-14T05:40:11.246 INFO:teuthology.orchestra.run.smithi194.stdout: 27 496 10.1957 41 KiB/s 2024-09-14T05:40:12.621 INFO:teuthology.orchestra.run.smithi194.stdout: 29 512 10.5035 42 KiB/s 2024-09-14T05:40:13.739 INFO:teuthology.orchestra.run.smithi194.stdout: 30 528 12.4938 50 KiB/s 2024-09-14T05:40:15.003 INFO:teuthology.orchestra.run.smithi194.stdout: 31 592 18.0546 72 KiB/s 2024-09-14T05:40:17.878 INFO:teuthology.orchestra.run.smithi194.stdout: 34 640 20.7748 83 KiB/s 2024-09-14T05:40:18.711 INFO:teuthology.orchestra.run.smithi194.stdout: 35 656 21.4367 86 KiB/s 2024-09-14T05:40:21.327 INFO:teuthology.orchestra.run.smithi194.stdout: 37 688 20.2118 81 KiB/s 2024-09-14T05:40:25.860 INFO:teuthology.orchestra.run.smithi194.stdout: 42 704 14.5218 58 KiB/s 2024-09-14T05:40:26.626 INFO:teuthology.orchestra.run.smithi194.stdout: 43 736 12.3927 50 KiB/s 2024-09-14T05:40:27.593 INFO:teuthology.orchestra.run.smithi194.stdout: 44 768 13.1745 53 KiB/s 2024-09-14T05:40:28.676 INFO:teuthology.orchestra.run.smithi194.stdout: 45 784 12.8414 51 KiB/s 2024-09-14T05:40:29.901 INFO:teuthology.orchestra.run.smithi194.stdout: 46 832 16.7993 67 KiB/s 2024-09-14T05:40:31.351 INFO:teuthology.orchestra.run.smithi194.stdout: 47 864 29.1128 116 KiB/s 2024-09-14T05:40:32.100 INFO:teuthology.orchestra.run.smithi194.stdout: 48 880 26.3164 105 KiB/s 2024-09-14T05:40:34.251 INFO:teuthology.orchestra.run.smithi194.stdout: 50 912 21.6221 86 KiB/s 2024-09-14T05:40:35.100 INFO:teuthology.orchestra.run.smithi194.stdout: 51 928 22.4304 90 KiB/s 2024-09-14T05:40:38.724 INFO:teuthology.orchestra.run.smithi194.stdout: 55 944 12.6929 51 KiB/s 2024-09-14T05:40:39.774 INFO:teuthology.orchestra.run.smithi194.stdout: 56 976 13.302 53 KiB/s 2024-09-14T05:40:41.316 INFO:teuthology.orchestra.run.smithi194.stdout: 57 1024 15.6254 63 KiB/s 2024-09-14T05:40:42.228 INFO:teuthology.orchestra.run.smithi194.stdout: 58 1040 16.0566 64 KiB/s 2024-09-14T05:40:44.715 INFO:teuthology.orchestra.run.smithi194.stdout: 61 1072 14.9754 60 KiB/s 2024-09-14T05:40:46.081 INFO:teuthology.orchestra.run.smithi194.stdout: 62 1088 19.5657 78 KiB/s 2024-09-14T05:40:46.656 INFO:teuthology.orchestra.run.smithi194.stdout: 63 1104 18.6051 74 KiB/s 2024-09-14T05:40:47.831 INFO:teuthology.orchestra.run.smithi194.stdout: 64 1168 22.0999 88 KiB/s 2024-09-14T05:40:48.588 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:40:48.588 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:40:48.597 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:41:28.598 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:41:28.599 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:41:28.657 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:40:48 2024","present":true,"empty":false,"clean":false,"allocated_bytes":491659264,"cached_bytes":491659264,"dirty_bytes":667648,"free_bytes":259960012,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:41:28.659 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:41:28.659 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:41:28.663 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:41:28.663 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern rand --io-size 1024K --io-total 100G testimage 2024-09-14T05:41:29.389 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 1048576 io_threads 16 bytes 107374182400 pattern random 2024-09-14T05:41:33.861 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:41:33.862 INFO:teuthology.orchestra.run.smithi194.stdout: 4 32 86.9588 87 MiB/s 2024-09-14T05:41:34.641 INFO:teuthology.orchestra.run.smithi194.stdout: 5 48 48.194 48 MiB/s 2024-09-14T05:41:35.585 INFO:teuthology.orchestra.run.smithi194.stdout: 6 64 35.2122 35 MiB/s 2024-09-14T05:41:37.060 INFO:teuthology.orchestra.run.smithi194.stdout: 7 96 29.8834 30 MiB/s 2024-09-14T05:41:38.169 INFO:teuthology.orchestra.run.smithi194.stdout: 8 112 26.3381 26 MiB/s 2024-09-14T05:41:38.707 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:41:38.707 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:41:38.764 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/580768066","client":4517,"cookie":140220750140560}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:41:34 2024","present":true,"empty":false,"clean":false,"allocated_bytes":539484160,"cached_bytes":539484160,"dirty_bytes":52531200,"free_bytes":212135116,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:41:38.765 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:41:39.264 INFO:teuthology.orchestra.run.smithi194.stdout: 9 128 17.7782 18 MiB/s 2024-09-14T05:41:40.122 INFO:teuthology.orchestra.run.smithi194.stdout: 10 144 17.5059 18 MiB/s 2024-09-14T05:41:42.103 INFO:teuthology.orchestra.run.smithi194.stdout: 12 160 14.7243 15 MiB/s 2024-09-14T05:41:43.380 INFO:teuthology.orchestra.run.smithi194.stdout: 13 176 12.6586 13 MiB/s 2024-09-14T05:41:44.407 INFO:teuthology.orchestra.run.smithi194.stdout: 15 192 12.8291 13 MiB/s 2024-09-14T05:41:47.305 INFO:teuthology.orchestra.run.smithi194.stdout: 17 224 11.9406 12 MiB/s 2024-09-14T05:41:48.314 INFO:teuthology.orchestra.run.smithi194.stdout: 18 240 11.719 12 MiB/s 2024-09-14T05:41:49.214 INFO:teuthology.orchestra.run.smithi194.stdout: 19 256 13.4987 13 MiB/s 2024-09-14T05:41:53.953 INFO:teuthology.orchestra.run.smithi194.stdout: 24 272 9.08082 9.1 MiB/s 2024-09-14T05:42:13.105 INFO:teuthology.orchestra.run.smithi194.stdout: 43 288 3.3455 3.3 MiB/s 2024-09-14T05:42:22.055 INFO:teuthology.orchestra.run.smithi194.stdout: 52 304 2.30208 2.3 MiB/s 2024-09-14T05:42:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: 54 320 2.24598 2.2 MiB/s 2024-09-14T05:42:27.605 INFO:teuthology.orchestra.run.smithi194.stdout: 58 336 2.08382 2.1 MiB/s 2024-09-14T05:42:28.997 INFO:teuthology.orchestra.run.smithi194.stdout: 59 352 2.2829 2.3 MiB/s 2024-09-14T05:42:30.305 INFO:teuthology.orchestra.run.smithi194.stdout: 60 368 4.65019 4.7 MiB/s 2024-09-14T05:42:31.565 INFO:teuthology.orchestra.run.smithi194.stdout: 62 384 8.41417 8.4 MiB/s 2024-09-14T05:42:32.766 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:42:32.767 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:42:32.777 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:43:12.841 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:43:12.842 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:43:12.902 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:42:29 2024","present":true,"empty":false,"clean":false,"allocated_bytes":750252032,"cached_bytes":750252032,"dirty_bytes":6291456,"free_bytes":1367244,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:43:12.904 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:43:12.905 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:43:12.909 INFO:tasks.rbd_pwl_cache_recovery:start rbd bench 2024-09-14T05:43:12.909 DEBUG:teuthology.orchestra.run.smithi194:> rbd bench --io-type write --io-pattern full-seq --io-size 1024K --io-total 100G testimage 2024-09-14T05:43:13.452 INFO:teuthology.orchestra.run.smithi194.stdout:bench type write io_size 1048576 io_threads 16 bytes 107374182400 pattern full sequential 2024-09-14T05:43:16.194 INFO:teuthology.orchestra.run.smithi194.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-14T05:43:16.194 INFO:teuthology.orchestra.run.smithi194.stdout: 2 32 36.8106 37 MiB/s 2024-09-14T05:43:17.658 INFO:teuthology.orchestra.run.smithi194.stdout: 4 48 23.1219 23 MiB/s 2024-09-14T05:43:18.914 INFO:teuthology.orchestra.run.smithi194.stdout: 5 64 19.8812 20 MiB/s 2024-09-14T05:43:20.416 INFO:teuthology.orchestra.run.smithi194.stdout: 6 80 17.3791 17 MiB/s 2024-09-14T05:43:21.832 INFO:teuthology.orchestra.run.smithi194.stdout: 8 96 16.1387 16 MiB/s 2024-09-14T05:43:22.955 INFO:tasks.rbd_pwl_cache_recovery:dump cache state when rbd bench running. 2024-09-14T05:43:22.956 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:43:23.015 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[{"address":"172.21.15.194:0/2246909093","client":4531,"cookie":139813533553616}],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:43:18 2024","present":true,"empty":false,"clean":false,"allocated_bytes":750776320,"cached_bytes":750776320,"dirty_bytes":7340032,"free_bytes":842956,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:43:23.017 INFO:tasks.rbd_pwl_cache_recovery:sleep... 2024-09-14T05:43:23.707 INFO:teuthology.orchestra.run.smithi194.stdout: 10 112 10.6499 11 MiB/s 2024-09-14T05:43:27.804 INFO:teuthology.orchestra.run.smithi194.stdout: 14 128 7.88661 7.9 MiB/s 2024-09-14T05:43:29.189 INFO:teuthology.orchestra.run.smithi194.stdout: 15 144 7.7853 7.8 MiB/s 2024-09-14T05:43:30.572 INFO:teuthology.orchestra.run.smithi194.stdout: 17 160 7.87729 7.9 MiB/s 2024-09-14T05:43:32.119 INFO:teuthology.orchestra.run.smithi194.stdout: 18 176 7.77622 7.8 MiB/s 2024-09-14T05:43:33.745 INFO:teuthology.orchestra.run.smithi194.stdout: 20 192 7.97148 8.0 MiB/s 2024-09-14T05:43:36.819 INFO:teuthology.orchestra.run.smithi194.stdout: 23 208 8.8733 8.9 MiB/s 2024-09-14T05:43:37.018 INFO:tasks.rbd_pwl_cache_recovery:rbd bench crash. 2024-09-14T05:43:37.018 DEBUG:teuthology.orchestra.run.smithi194:> killall -9 rbd 2024-09-14T05:43:37.028 INFO:tasks.rbd_pwl_cache_recovery:wait for watch timeout. 2024-09-14T05:44:17.029 INFO:tasks.rbd_pwl_cache_recovery:check cache state after crash. 2024-09-14T05:44:17.029 DEBUG:teuthology.orchestra.run.smithi194:> rbd status testimage --format=json 2024-09-14T05:44:17.093 INFO:teuthology.orchestra.run.smithi194.stdout:{"watchers":[],"persistent_cache":{"host":"smithi194","path":"/home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool","size":1073741824,"mode":"rwl","stats_timestamp":"Sat Sep 14 05:43:33 2024","present":true,"empty":false,"clean":false,"allocated_bytes":746582016,"cached_bytes":746582016,"dirty_bytes":3145728,"free_bytes":5037260,"hits_full":0,"hits_full_percent":0,"hits_partial":0,"hits_partial_percent":0,"misses":0,"hit_bytes":0,"hit_bytes_percent":0,"miss_bytes":0}} 2024-09-14T05:44:17.095 INFO:tasks.rbd_pwl_cache_recovery:check dirty cache file. 2024-09-14T05:44:17.095 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/rbd-pwl-cache/rbd-pwl.rbd.10a393d80590.pool 2024-09-14T05:44:17.100 DEBUG:teuthology.run_tasks:Unwinding manager rbd_pwl_cache_recovery 2024-09-14T05:44:17.111 INFO:tasks.rbd_pwl_cache_recovery:cleanup 2024-09-14T05:44:17.111 DEBUG:teuthology.run_tasks:Unwinding manager rbd.create_image 2024-09-14T05:44:17.119 INFO:tasks.rbd:Deleting rbd images... 2024-09-14T05:44:17.120 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/passphrase 2024-09-14T05:44:17.145 DEBUG:teuthology.orchestra.run.smithi194:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd --cluster ceph -p rbd rm testimage 2024-09-14T05:44:29.666 INFO:teuthology.orchestra.run.smithi194.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T05:44:29.677 DEBUG:teuthology.run_tasks:Unwinding manager exec_on_cleanup 2024-09-14T05:44:29.687 INFO:tasks.exec_on_cleanup:Executing custom commands... 2024-09-14T05:44:29.688 INFO:tasks.exec_on_cleanup:Running commands on role client.0 host ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:44:29.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rm -rf /home/ubuntu/cephtest/rbd-pwl-cache' 2024-09-14T05:44:29.703 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T05:44:29.715 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-14T05:44:29.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:44:30.024 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:30.025 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:44:30.042 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":966,"stamp":"2024-09-14T05:44:28.184681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9659765989,"num_objects":2568,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":21777,"num_read_kb":18349,"num_write":33592,"num_write_kb":2544585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31087,"ondisk_log_size":31087,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":4006320,"kb_used_data":3741352,"kb_used_omap":6,"kb_used_meta":264889,"kb_avail":415424080,"statfs":{"total":429496729600,"available":425394257920,"internally_reserved":0,"allocated":3831144448,"data_stored":3830317050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":271247126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":393216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":93,"num_write":30,"num_write_kb":11274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.923380"},"pg_stats":[{"pgid":"2.7","version":"36'3244","reported_seq":3320,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262536+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:44:17.262536+0000","last_peered":"2024-09-14T05:44:17.262536+0000","last_clean":"2024-09-14T05:44:17.262536+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:44:17.262536+0000","last_undegraded":"2024-09-14T05:44:17.262536+0000","last_fullsized":"2024-09-14T05:44:17.262536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":1150001152,"num_objects":304,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":332196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"36'3500","reported_seq":3576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262289+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:44:17.262289+0000","last_peered":"2024-09-14T05:44:17.262289+0000","last_clean":"2024-09-14T05:44:17.262289+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:44:17.262289+0000","last_undegraded":"2024-09-14T05:44:17.262289+0000","last_fullsized":"2024-09-14T05:44:17.262289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":1263312896,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":329432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"37'3049","reported_seq":3125,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.649584+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:44:17.649584+0000","last_peered":"2024-09-14T05:44:17.649584+0000","last_clean":"2024-09-14T05:44:17.649584+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:44:17.649584+0000","last_undegraded":"2024-09-14T05:44:17.649584+0000","last_fullsized":"2024-09-14T05:44:17.649584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":1204756480,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":261508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"37'7023","reported_seq":7265,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.057105+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:44:26.057105+0000","last_peered":"2024-09-14T05:44:26.057105+0000","last_clean":"2024-09-14T05:44:26.057105+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:44:26.057105+0000","last_undegraded":"2024-09-14T05:44:26.057105+0000","last_fullsized":"2024-09-14T05:44:26.057105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":7023,"log_dups_size":0,"ondisk_log_size":7023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":1217733298,"num_objects":326,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":15655,"num_read_kb":12949,"num_write":9563,"num_write_kb":381599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'3325","reported_seq":3716,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.692126+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:44:17.692126+0000","last_peered":"2024-09-14T05:44:17.692126+0000","last_clean":"2024-09-14T05:44:17.692126+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:44:17.692126+0000","last_undegraded":"2024-09-14T05:44:17.692126+0000","last_fullsized":"2024-09-14T05:44:17.692126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":1202618387,"num_objects":323,"num_object_clones":0,"num_object_copies":646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":315,"num_read_kb":315,"num_write":3325,"num_write_kb":334446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"36'3495","reported_seq":3571,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262326+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:44:17.262326+0000","last_peered":"2024-09-14T05:44:17.262326+0000","last_clean":"2024-09-14T05:44:17.262326+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:44:17.262326+0000","last_undegraded":"2024-09-14T05:44:17.262326+0000","last_fullsized":"2024-09-14T05:44:17.262326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":1096527872,"num_objects":294,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":282428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3730","reported_seq":6193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.046774+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:44:26.046774+0000","last_peered":"2024-09-14T05:44:26.046774+0000","last_clean":"2024-09-14T05:44:26.046774+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:44:26.046774+0000","last_undegraded":"2024-09-14T05:44:26.046774+0000","last_fullsized":"2024-09-14T05:44:26.046774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":1262108672,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":5741,"num_read_kb":5030,"num_write":3667,"num_write_kb":312764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"36'3649","reported_seq":3725,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262486+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:44:17.262486+0000","last_peered":"2024-09-14T05:44:17.262486+0000","last_clean":"2024-09-14T05:44:17.262486+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:44:17.262486+0000","last_undegraded":"2024-09-14T05:44:17.262486+0000","last_fullsized":"2024-09-14T05:44:17.262486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":1262116864,"num_objects":335,"num_object_clones":0,"num_object_copies":670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":308584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262472+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:44:17.262472+0000","last_peered":"2024-09-14T05:44:17.262472+0000","last_clean":"2024-09-14T05:44:17.262472+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:44:17.262472+0000","last_undegraded":"2024-09-14T05:44:17.262472+0000","last_fullsized":"2024-09-14T05:44:17.262472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9659175621,"num_objects":2566,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":21711,"num_read_kb":18294,"num_write":33491,"num_write_kb":2542957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3828629504,"data_stored":3828606346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31015,"ondisk_log_size":31015,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673335,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1224424,"kb_used_data":1179352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":103633176,"statfs":{"total":107374182400,"available":106120372224,"internally_reserved":0,"allocated":1207656448,"data_stored":1207449535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1253380,"kb_used_data":1168052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":103604220,"statfs":{"total":107374182400,"available":106090721280,"internally_reserved":0,"allocated":1196085248,"data_stored":1195874385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706038,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":774052,"kb_used_data":724816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104083548,"statfs":{"total":107374182400,"available":106581553152,"internally_reserved":0,"allocated":742211584,"data_stored":742012300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706039,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":754464,"kb_used_data":669132,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":104103136,"statfs":{"total":107374182400,"available":106601611264,"internally_reserved":0,"allocated":685191168,"data_stored":684980830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":684265472,"data_stored":684257970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1195753472,"data_stored":1195741893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":741879808,"data_stored":741879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1206730752,"data_stored":1206726675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:44:30.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:44:30.360 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:30.361 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:44:30.379 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":967,"stamp":"2024-09-14T05:44:30.185129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9659765989,"num_objects":2568,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":21777,"num_read_kb":18349,"num_write":33592,"num_write_kb":2544585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31087,"ondisk_log_size":31087,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":3188172,"kb_used_data":2923212,"kb_used_omap":6,"kb_used_meta":264889,"kb_avail":416242228,"statfs":{"total":429496729600,"available":426232041472,"internally_reserved":0,"allocated":2993369088,"data_stored":2992545770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":271247126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":393216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":93,"num_write":30,"num_write_kb":11274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"2.7","version":"36'3244","reported_seq":3320,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262536+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:44:17.262536+0000","last_peered":"2024-09-14T05:44:17.262536+0000","last_clean":"2024-09-14T05:44:17.262536+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:44:17.262536+0000","last_undegraded":"2024-09-14T05:44:17.262536+0000","last_fullsized":"2024-09-14T05:44:17.262536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":1150001152,"num_objects":304,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":332196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"36'3500","reported_seq":3576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262289+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:44:17.262289+0000","last_peered":"2024-09-14T05:44:17.262289+0000","last_clean":"2024-09-14T05:44:17.262289+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:44:17.262289+0000","last_undegraded":"2024-09-14T05:44:17.262289+0000","last_fullsized":"2024-09-14T05:44:17.262289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":1263312896,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":329432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"37'3049","reported_seq":3125,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.649584+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:44:17.649584+0000","last_peered":"2024-09-14T05:44:17.649584+0000","last_clean":"2024-09-14T05:44:17.649584+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:44:17.649584+0000","last_undegraded":"2024-09-14T05:44:17.649584+0000","last_fullsized":"2024-09-14T05:44:17.649584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":1204756480,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":261508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"37'7023","reported_seq":7265,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.057105+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:44:26.057105+0000","last_peered":"2024-09-14T05:44:26.057105+0000","last_clean":"2024-09-14T05:44:26.057105+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:44:26.057105+0000","last_undegraded":"2024-09-14T05:44:26.057105+0000","last_fullsized":"2024-09-14T05:44:26.057105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":7023,"log_dups_size":0,"ondisk_log_size":7023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":1217733298,"num_objects":326,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":15655,"num_read_kb":12949,"num_write":9563,"num_write_kb":381599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'3325","reported_seq":3716,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.692126+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:44:17.692126+0000","last_peered":"2024-09-14T05:44:17.692126+0000","last_clean":"2024-09-14T05:44:17.692126+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:44:17.692126+0000","last_undegraded":"2024-09-14T05:44:17.692126+0000","last_fullsized":"2024-09-14T05:44:17.692126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":1202618387,"num_objects":323,"num_object_clones":0,"num_object_copies":646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":315,"num_read_kb":315,"num_write":3325,"num_write_kb":334446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"36'3495","reported_seq":3571,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262326+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:44:17.262326+0000","last_peered":"2024-09-14T05:44:17.262326+0000","last_clean":"2024-09-14T05:44:17.262326+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:44:17.262326+0000","last_undegraded":"2024-09-14T05:44:17.262326+0000","last_fullsized":"2024-09-14T05:44:17.262326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":1096527872,"num_objects":294,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":282428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3730","reported_seq":6193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.046774+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:44:26.046774+0000","last_peered":"2024-09-14T05:44:26.046774+0000","last_clean":"2024-09-14T05:44:26.046774+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:44:26.046774+0000","last_undegraded":"2024-09-14T05:44:26.046774+0000","last_fullsized":"2024-09-14T05:44:26.046774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":1262108672,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":5741,"num_read_kb":5030,"num_write":3667,"num_write_kb":312764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"36'3649","reported_seq":3725,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262486+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:44:17.262486+0000","last_peered":"2024-09-14T05:44:17.262486+0000","last_clean":"2024-09-14T05:44:17.262486+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:44:17.262486+0000","last_undegraded":"2024-09-14T05:44:17.262486+0000","last_fullsized":"2024-09-14T05:44:17.262486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":1262116864,"num_objects":335,"num_object_clones":0,"num_object_copies":670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":308584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262472+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:44:17.262472+0000","last_peered":"2024-09-14T05:44:17.262472+0000","last_clean":"2024-09-14T05:44:17.262472+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:44:17.262472+0000","last_undegraded":"2024-09-14T05:44:17.262472+0000","last_fullsized":"2024-09-14T05:44:17.262472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9659175621,"num_objects":2566,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":21711,"num_read_kb":18294,"num_write":33491,"num_write_kb":2542957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2402484224,"data_stored":2402468568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31015,"ondisk_log_size":31015,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673335,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1224424,"kb_used_data":1179352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":103633176,"statfs":{"total":107374182400,"available":106120372224,"internally_reserved":0,"allocated":1207656448,"data_stored":1207449535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1253380,"kb_used_data":1168052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":103604220,"statfs":{"total":107374182400,"available":106090721280,"internally_reserved":0,"allocated":1196085248,"data_stored":1195874385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706039,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":389880,"kb_used_data":340648,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104467720,"statfs":{"total":107374182400,"available":106974945280,"internally_reserved":0,"allocated":348823552,"data_stored":348624268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706040,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":320488,"kb_used_data":235160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":104537112,"statfs":{"total":107374182400,"available":107046002688,"internally_reserved":0,"allocated":240803840,"data_stored":240597582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1195753472,"data_stored":1195741893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1206730752,"data_stored":1206726675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:44:30.380 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-14T05:44:30.380 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:44:30.671 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:30.671 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:44:30.687 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":967,"stamp":"2024-09-14T05:44:30.185129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9659765989,"num_objects":2568,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":21777,"num_read_kb":18349,"num_write":33592,"num_write_kb":2544585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31087,"ondisk_log_size":31087,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":3188172,"kb_used_data":2923212,"kb_used_omap":6,"kb_used_meta":264889,"kb_avail":416242228,"statfs":{"total":429496729600,"available":426232041472,"internally_reserved":0,"allocated":2993369088,"data_stored":2992545770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":271247126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":393216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":93,"num_write":30,"num_write_kb":11274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"2.7","version":"36'3244","reported_seq":3320,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262536+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:44:17.262536+0000","last_peered":"2024-09-14T05:44:17.262536+0000","last_clean":"2024-09-14T05:44:17.262536+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:44:17.262536+0000","last_undegraded":"2024-09-14T05:44:17.262536+0000","last_fullsized":"2024-09-14T05:44:17.262536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":1150001152,"num_objects":304,"num_object_clones":0,"num_object_copies":608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":332196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"36'3500","reported_seq":3576,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262289+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:44:17.262289+0000","last_peered":"2024-09-14T05:44:17.262289+0000","last_clean":"2024-09-14T05:44:17.262289+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:44:17.262289+0000","last_undegraded":"2024-09-14T05:44:17.262289+0000","last_fullsized":"2024-09-14T05:44:17.262289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":1263312896,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3500,"num_write_kb":329432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"37'3049","reported_seq":3125,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.649584+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:44:17.649584+0000","last_peered":"2024-09-14T05:44:17.649584+0000","last_clean":"2024-09-14T05:44:17.649584+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:44:17.649584+0000","last_undegraded":"2024-09-14T05:44:17.649584+0000","last_fullsized":"2024-09-14T05:44:17.649584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":1204756480,"num_objects":320,"num_object_clones":0,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":261508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"37'7023","reported_seq":7265,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.057105+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:44:26.057105+0000","last_peered":"2024-09-14T05:44:26.057105+0000","last_clean":"2024-09-14T05:44:26.057105+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:44:26.057105+0000","last_undegraded":"2024-09-14T05:44:26.057105+0000","last_fullsized":"2024-09-14T05:44:26.057105+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":7023,"log_dups_size":0,"ondisk_log_size":7023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":1217733298,"num_objects":326,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":15655,"num_read_kb":12949,"num_write":9563,"num_write_kb":381599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'3325","reported_seq":3716,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.692126+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:44:17.692126+0000","last_peered":"2024-09-14T05:44:17.692126+0000","last_clean":"2024-09-14T05:44:17.692126+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:44:17.692126+0000","last_undegraded":"2024-09-14T05:44:17.692126+0000","last_fullsized":"2024-09-14T05:44:17.692126+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":1202618387,"num_objects":323,"num_object_clones":0,"num_object_copies":646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":315,"num_read_kb":315,"num_write":3325,"num_write_kb":334446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"36'3495","reported_seq":3571,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262326+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:44:17.262326+0000","last_peered":"2024-09-14T05:44:17.262326+0000","last_clean":"2024-09-14T05:44:17.262326+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:44:17.262326+0000","last_undegraded":"2024-09-14T05:44:17.262326+0000","last_fullsized":"2024-09-14T05:44:17.262326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":1096527872,"num_objects":294,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":282428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'3730","reported_seq":6193,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:26.046774+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:44:26.046774+0000","last_peered":"2024-09-14T05:44:26.046774+0000","last_clean":"2024-09-14T05:44:26.046774+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:44:26.046774+0000","last_undegraded":"2024-09-14T05:44:26.046774+0000","last_fullsized":"2024-09-14T05:44:26.046774+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3730,"log_dups_size":0,"ondisk_log_size":3730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":1262108672,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":5741,"num_read_kb":5030,"num_write":3667,"num_write_kb":312764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"36'3649","reported_seq":3725,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262486+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:44:17.262486+0000","last_peered":"2024-09-14T05:44:17.262486+0000","last_clean":"2024-09-14T05:44:17.262486+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:44:17.262486+0000","last_undegraded":"2024-09-14T05:44:17.262486+0000","last_fullsized":"2024-09-14T05:44:17.262486+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":1262116864,"num_objects":335,"num_object_clones":0,"num_object_copies":670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3648,"num_write_kb":308584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262472+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:44:17.262472+0000","last_peered":"2024-09-14T05:44:17.262472+0000","last_clean":"2024-09-14T05:44:17.262472+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:44:17.262472+0000","last_undegraded":"2024-09-14T05:44:17.262472+0000","last_fullsized":"2024-09-14T05:44:17.262472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9659175621,"num_objects":2566,"num_object_clones":0,"num_object_copies":5132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":21711,"num_read_kb":18294,"num_write":33491,"num_write_kb":2542957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2402484224,"data_stored":2402468568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31015,"ondisk_log_size":31015,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673335,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1224424,"kb_used_data":1179352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":103633176,"statfs":{"total":107374182400,"available":106120372224,"internally_reserved":0,"allocated":1207656448,"data_stored":1207449535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1253380,"kb_used_data":1168052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":103604220,"statfs":{"total":107374182400,"available":106090721280,"internally_reserved":0,"allocated":1196085248,"data_stored":1195874385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706039,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":389880,"kb_used_data":340648,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104467720,"statfs":{"total":107374182400,"available":106974945280,"internally_reserved":0,"allocated":348823552,"data_stored":348624268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706040,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":320488,"kb_used_data":235160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":104537112,"statfs":{"total":107374182400,"available":107046002688,"internally_reserved":0,"allocated":240803840,"data_stored":240597582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1195753472,"data_stored":1195741893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1206730752,"data_stored":1206726675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:44:30.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-14T05:44:30.974 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:30.975 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":37,"fsid":"aaa809bf-6549-4b19-b0ac-70959376c827","created":"2024-09-14T05:13:16.899381+0000","modified":"2024-09-14T05:44:17.257997+0000","last_up_change":"2024-09-14T05:13:26.503913+0000","last_in_change":"2024-09-14T05:13:18.100075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:13:25.740988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:13:32.715764+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8387a05-b5c5-4510-baa7-ae9d47d0fde6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6801","nonce":736494405}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6803","nonce":736494405}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6807","nonce":736494405}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":736494405},{"type":"v1","addr":"172.21.15.144:6805","nonce":736494405}]},"public_addr":"172.21.15.144:6801/736494405","cluster_addr":"172.21.15.144:6803/736494405","heartbeat_back_addr":"172.21.15.144:6807/736494405","heartbeat_front_addr":"172.21.15.144:6805/736494405","state":["exists","up"]},{"osd":1,"uuid":"0908a852-a99b-4d23-ae6f-65ed6b425bfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6809","nonce":2747334951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6811","nonce":2747334951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6815","nonce":2747334951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":2747334951},{"type":"v1","addr":"172.21.15.144:6813","nonce":2747334951}]},"public_addr":"172.21.15.144:6809/2747334951","cluster_addr":"172.21.15.144:6811/2747334951","heartbeat_back_addr":"172.21.15.144:6815/2747334951","heartbeat_front_addr":"172.21.15.144:6813/2747334951","state":["exists","up"]},{"osd":2,"uuid":"c0ce7c85-9863-4713-8659-aeabca49bbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6801","nonce":520007229}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6803","nonce":520007229}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6807","nonce":520007229}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":520007229},{"type":"v1","addr":"172.21.15.194:6805","nonce":520007229}]},"public_addr":"172.21.15.194:6801/520007229","cluster_addr":"172.21.15.194:6803/520007229","heartbeat_back_addr":"172.21.15.194:6807/520007229","heartbeat_front_addr":"172.21.15.194:6805/520007229","state":["exists","up"]},{"osd":3,"uuid":"0e7cbc8f-59a1-48fa-b846-8f34c34f8f9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6809","nonce":796375837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6811","nonce":796375837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6815","nonce":796375837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":796375837},{"type":"v1","addr":"172.21.15.194:6813","nonce":796375837}]},"public_addr":"172.21.15.194:6809/796375837","cluster_addr":"172.21.15.194:6811/796375837","heartbeat_back_addr":"172.21.15.194:6815/796375837","heartbeat_front_addr":"172.21.15.194:6813/796375837","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.664216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.993903+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.627468+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:13:23.788647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.194:0/2246909093":"2024-09-14T06:44:17.257767+0000","172.21.15.194:0/580768066":"2024-09-14T06:43:13.027869+0000","172.21.15.194:0/3253393924":"2024-09-14T06:39:42.597518+0000","172.21.15.194:0/77466580":"2024-09-14T06:41:28.777919+0000","172.21.15.194:0/1112762793":"2024-09-14T06:37:23.219680+0000","172.21.15.194:0/3168896081":"2024-09-14T06:36:14.039068+0000","172.21.15.194:0/3136032906":"2024-09-14T06:14:57.621428+0000","172.21.15.194:0/544481073":"2024-09-14T06:28:22.123356+0000","172.21.15.194:0/2595284434":"2024-09-14T06:16:26.815134+0000","172.21.15.194:0/1806925821":"2024-09-14T06:21:54.376821+0000","172.21.15.194:0/2571992308":"2024-09-14T06:18:17.016794+0000","172.21.15.194:0/2588195778":"2024-09-14T06:38:28.408803+0000","172.21.15.194:0/4031237887":"2024-09-14T06:23:37.565964+0000","172.21.15.194:0/3684106762":"2024-09-14T06:25:01.755926+0000","172.21.15.194:0/4174748124":"2024-09-14T06:26:42.934490+0000","172.21.15.194:0/1911006792":"2024-09-14T06:33:00.662309+0000","172.21.15.194:0/3019347368":"2024-09-14T06:31:18.482855+0000","172.21.15.194:0/1870280846":"2024-09-14T06:20:07.196620+0000","172.21.15.194:0/3854267793":"2024-09-14T06:29:51.302869+0000","172.21.15.194:0/1434599906":"2024-09-14T06:34:27.852625+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:44:31.993 INFO:tasks.ceph:Scrubbing osd.0 2024-09-14T05:44:31.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T05:44:32.154 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-14T05:44:32.154 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T05:44:32.154 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-14T05:44:32.170 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-14T05:44:32.465 INFO:teuthology.orchestra.run.smithi144.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T05:44:32.483 INFO:tasks.ceph:Scrubbing osd.1 2024-09-14T05:44:32.483 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T05:44:32.617 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-14T05:44:32.617 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T05:44:32.617 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-14T05:44:32.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-14T05:44:32.911 INFO:teuthology.orchestra.run.smithi144.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T05:44:32.928 INFO:tasks.ceph:Scrubbing osd.2 2024-09-14T05:44:32.928 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T05:44:33.062 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-14T05:44:33.063 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T05:44:33.063 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-14T05:44:33.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-14T05:44:33.354 INFO:teuthology.orchestra.run.smithi144.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T05:44:33.371 INFO:tasks.ceph:Scrubbing osd.3 2024-09-14T05:44:33.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-14T05:44:33.506 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-14T05:44:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T05:44:33.507 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-14T05:44:33.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-14T05:44:33.793 INFO:teuthology.orchestra.run.smithi144.stderr:instructed osd(s) 3 to deep-scrub 2024-09-14T05:44:33.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:44:34.081 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:34.081 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:44:34.097 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":968,"stamp":"2024-09-14T05:44:32.185438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21837,"num_read_kb":18403,"num_write":36187,"num_write_kb":2544606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33684,"ondisk_log_size":33684,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":850328,"kb_used_data":577044,"kb_used_omap":6,"kb_used_meta":273209,"kb_avail":418580072,"statfs":{"total":429496729600,"available":428625993728,"internally_reserved":0,"allocated":590893056,"data_stored":590077240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":279766806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9658782386,"num_objects":-2563,"num_object_clones":0,"num_object_copies":-5126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2563,"num_whiteouts":0,"num_read":167,"num_read_kb":147,"num_write":2625,"num_write_kb":11295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002775"},"pg_stats":[{"pgid":"2.7","version":"37'3548","reported_seq":3624,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.581463+0000","last_change":"2024-09-14T05:13:36.589135+0000","last_active":"2024-09-14T05:44:29.581463+0000","last_peered":"2024-09-14T05:44:29.581463+0000","last_clean":"2024-09-14T05:44:29.581463+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:44:29.581463+0000","last_undegraded":"2024-09-14T05:44:29.581463+0000","last_fullsized":"2024-09-14T05:44:29.581463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:25:28.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":332196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"37'3833","reported_seq":3909,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.587330+0000","last_change":"2024-09-14T05:13:36.583298+0000","last_active":"2024-09-14T05:44:29.587330+0000","last_peered":"2024-09-14T05:44:29.587330+0000","last_clean":"2024-09-14T05:44:29.587330+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:44:29.587330+0000","last_undegraded":"2024-09-14T05:44:29.587330+0000","last_fullsized":"2024-09-14T05:44:29.587330+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:05.877463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":329432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"37'3369","reported_seq":3445,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.585807+0000","last_change":"2024-09-14T05:13:36.588684+0000","last_active":"2024-09-14T05:44:29.585807+0000","last_peered":"2024-09-14T05:44:29.585807+0000","last_clean":"2024-09-14T05:44:29.585807+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:44:29.585807+0000","last_undegraded":"2024-09-14T05:44:29.585807+0000","last_fullsized":"2024-09-14T05:44:29.585807+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:58:11.616142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":261508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"37'7354","reported_seq":7597,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.638114+0000","last_change":"2024-09-14T05:13:36.582987+0000","last_active":"2024-09-14T05:44:29.638114+0000","last_peered":"2024-09-14T05:44:29.638114+0000","last_clean":"2024-09-14T05:44:29.638114+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:44:29.638114+0000","last_undegraded":"2024-09-14T05:44:29.638114+0000","last_fullsized":"2024-09-14T05:44:29.638114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:37.581697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15674,"num_read_kb":12965,"num_write":9897,"num_write_kb":381603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'3647","reported_seq":4039,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.583513+0000","last_change":"2024-09-14T05:13:36.590673+0000","last_active":"2024-09-14T05:44:29.583513+0000","last_peered":"2024-09-14T05:44:29.583513+0000","last_clean":"2024-09-14T05:44:29.583513+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:44:29.583513+0000","last_undegraded":"2024-09-14T05:44:29.583513+0000","last_fullsized":"2024-09-14T05:44:29.583513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:22.538765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":316,"num_read_kb":316,"num_write":3647,"num_write_kb":334446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"37'3789","reported_seq":3865,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.578014+0000","last_change":"2024-09-14T05:13:36.582721+0000","last_active":"2024-09-14T05:44:29.578014+0000","last_peered":"2024-09-14T05:44:29.578014+0000","last_clean":"2024-09-14T05:44:29.578014+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:44:29.578014+0000","last_undegraded":"2024-09-14T05:44:29.578014+0000","last_fullsized":"2024-09-14T05:44:29.578014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:35:36.197746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":282428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4085","reported_seq":6592,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.647801+0000","last_change":"2024-09-14T05:13:36.588563+0000","last_active":"2024-09-14T05:44:29.647801+0000","last_peered":"2024-09-14T05:44:29.647801+0000","last_clean":"2024-09-14T05:44:29.647801+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:44:29.647801+0000","last_undegraded":"2024-09-14T05:44:29.647801+0000","last_fullsized":"2024-09-14T05:44:29.647801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:03.191519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5775,"num_read_kb":5061,"num_write":4018,"num_write_kb":312778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"37'3987","reported_seq":4067,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:29.667865+0000","last_change":"2024-09-14T05:13:36.589331+0000","last_active":"2024-09-14T05:44:29.667865+0000","last_peered":"2024-09-14T05:44:29.667865+0000","last_clean":"2024-09-14T05:44:29.667865+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:44:29.667865+0000","last_undegraded":"2024-09-14T05:44:29.667865+0000","last_fullsized":"2024-09-14T05:44:29.667865+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:33.542846+0000","last_clean_scrub_stamp":"2024-09-14T05:13:33.542846+0000","objects_scrubbed":0,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:30:56.024720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3986,"num_write_kb":308587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'72","reported_seq":168,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:17.262472+0000","last_change":"2024-09-14T05:13:27.696637+0000","last_active":"2024-09-14T05:44:17.262472+0000","last_peered":"2024-09-14T05:44:17.262472+0000","last_clean":"2024-09-14T05:44:17.262472+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:44:17.262472+0000","last_undegraded":"2024-09-14T05:44:17.262472+0000","last_fullsized":"2024-09-14T05:44:17.262472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:13:26.503913+0000","last_clean_scrub_stamp":"2024-09-14T05:13:26.503913+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:50:33.211135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21771,"num_read_kb":18348,"num_write":36086,"num_write_kb":2542978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33612,"ondisk_log_size":33612,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673336,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":50140,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104807460,"statfs":{"total":107374182400,"available":107322839040,"internally_reserved":0,"allocated":929792,"data_stored":722879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":89820,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":104767780,"statfs":{"total":107374182400,"available":107282206720,"internally_reserved":0,"allocated":335872,"data_stored":132511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706039,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":389880,"kb_used_data":340648,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104467720,"statfs":{"total":107374182400,"available":106974945280,"internally_reserved":0,"allocated":348823552,"data_stored":348624268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706040,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":320488,"kb_used_data":235160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":104537112,"statfs":{"total":107374182400,"available":107046002688,"internally_reserved":0,"allocated":240803840,"data_stored":240597582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.101 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.102 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.102 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T05:13:33.542846+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=33, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.102 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T05:13:26.503913+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=13, tm_sec=26, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=44, tm_sec=30, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T05:44:34.102 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T05:44:54.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-14T05:44:54.407 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-14T05:44:54.408 INFO:teuthology.orchestra.run.smithi144.stderr:dumped all 2024-09-14T05:44:54.424 INFO:teuthology.orchestra.run.smithi144.stdout:{"pg_ready":true,"pg_map":{"version":979,"stamp":"2024-09-14T05:44:54.190957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21837,"num_read_kb":18403,"num_write":36187,"num_write_kb":2544606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33687,"ondisk_log_size":33687,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":419430400,"kb_used":284064,"kb_used_data":2464,"kb_used_omap":6,"kb_used_meta":281529,"kb_avail":419146336,"statfs":{"total":429496729600,"available":429205848064,"internally_reserved":0,"allocated":2523136,"data_stored":1710742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":288286486},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"37'3548","reported_seq":3632,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:39.361750+0000","last_change":"2024-09-14T05:44:39.361750+0000","last_active":"2024-09-14T05:44:39.361750+0000","last_peered":"2024-09-14T05:44:39.361750+0000","last_clean":"2024-09-14T05:44:39.361750+0000","last_became_active":"2024-09-14T05:13:34.569149+0000","last_became_peered":"2024-09-14T05:13:34.569149+0000","last_unstale":"2024-09-14T05:44:39.361750+0000","last_undegraded":"2024-09-14T05:44:39.361750+0000","last_fullsized":"2024-09-14T05:44:39.361750+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3548","last_scrub_stamp":"2024-09-14T05:44:39.361608+0000","last_deep_scrub":"37'3548","last_deep_scrub_stamp":"2024-09-14T05:44:39.361608+0000","last_clean_scrub_stamp":"2024-09-14T05:44:39.361608+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:24.612503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001422172,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":332196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"37'3833","reported_seq":3917,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:33.196420+0000","last_change":"2024-09-14T05:44:33.196420+0000","last_active":"2024-09-14T05:44:33.196420+0000","last_peered":"2024-09-14T05:44:33.196420+0000","last_clean":"2024-09-14T05:44:33.196420+0000","last_became_active":"2024-09-14T05:13:34.566005+0000","last_became_peered":"2024-09-14T05:13:34.566005+0000","last_unstale":"2024-09-14T05:44:33.196420+0000","last_undegraded":"2024-09-14T05:44:33.196420+0000","last_fullsized":"2024-09-14T05:44:33.196420+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3833","last_scrub_stamp":"2024-09-14T05:44:33.196359+0000","last_deep_scrub":"37'3833","last_deep_scrub_stamp":"2024-09-14T05:44:33.196359+0000","last_clean_scrub_stamp":"2024-09-14T05:44:33.196359+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:02:47.641335+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001337537,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":329432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"37'3369","reported_seq":3453,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:38.362750+0000","last_change":"2024-09-14T05:44:38.362750+0000","last_active":"2024-09-14T05:44:38.362750+0000","last_peered":"2024-09-14T05:44:38.362750+0000","last_clean":"2024-09-14T05:44:38.362750+0000","last_became_active":"2024-09-14T05:13:34.572593+0000","last_became_peered":"2024-09-14T05:13:34.572593+0000","last_unstale":"2024-09-14T05:44:38.362750+0000","last_undegraded":"2024-09-14T05:44:38.362750+0000","last_fullsized":"2024-09-14T05:44:38.362750+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3369","last_scrub_stamp":"2024-09-14T05:44:38.362614+0000","last_deep_scrub":"37'3369","last_deep_scrub_stamp":"2024-09-14T05:44:38.362614+0000","last_clean_scrub_stamp":"2024-09-14T05:44:38.362614+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:38:03.883192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00097286900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3369,"num_write_kb":261508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"37'7354","reported_seq":7606,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:34.227856+0000","last_change":"2024-09-14T05:44:34.227620+0000","last_active":"2024-09-14T05:44:34.227856+0000","last_peered":"2024-09-14T05:44:34.227856+0000","last_clean":"2024-09-14T05:44:34.227856+0000","last_became_active":"2024-09-14T05:13:34.563849+0000","last_became_peered":"2024-09-14T05:13:34.563849+0000","last_unstale":"2024-09-14T05:44:34.227856+0000","last_undegraded":"2024-09-14T05:44:34.227856+0000","last_fullsized":"2024-09-14T05:44:34.227856+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'7354","last_scrub_stamp":"2024-09-14T05:44:34.227425+0000","last_deep_scrub":"37'7354","last_deep_scrub_stamp":"2024-09-14T05:44:34.227425+0000","last_clean_scrub_stamp":"2024-09-14T05:44:34.227425+0000","objects_scrubbed":1,"log_size":7354,"log_dups_size":0,"ondisk_log_size":7354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:31:40.792114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001023859,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15674,"num_read_kb":12965,"num_write":9897,"num_write_kb":381603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"37'3648","reported_seq":4049,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:37.355988+0000","last_change":"2024-09-14T05:44:37.355817+0000","last_active":"2024-09-14T05:44:37.355988+0000","last_peered":"2024-09-14T05:44:37.355988+0000","last_clean":"2024-09-14T05:44:37.355988+0000","last_became_active":"2024-09-14T05:13:34.572814+0000","last_became_peered":"2024-09-14T05:13:34.572814+0000","last_unstale":"2024-09-14T05:44:37.355988+0000","last_undegraded":"2024-09-14T05:44:37.355988+0000","last_fullsized":"2024-09-14T05:44:37.355988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3648","last_scrub_stamp":"2024-09-14T05:44:37.355678+0000","last_deep_scrub":"37'3648","last_deep_scrub_stamp":"2024-09-14T05:44:37.355678+0000","last_clean_scrub_stamp":"2024-09-14T05:44:37.355678+0000","objects_scrubbed":1,"log_size":3648,"log_dups_size":0,"ondisk_log_size":3648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:11.834060+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.002928347,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":316,"num_read_kb":316,"num_write":3647,"num_write_kb":334446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"37'3789","reported_seq":3874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:35.197257+0000","last_change":"2024-09-14T05:44:35.197122+0000","last_active":"2024-09-14T05:44:35.197257+0000","last_peered":"2024-09-14T05:44:35.197257+0000","last_clean":"2024-09-14T05:44:35.197257+0000","last_became_active":"2024-09-14T05:13:34.564094+0000","last_became_peered":"2024-09-14T05:13:34.564094+0000","last_unstale":"2024-09-14T05:44:35.197257+0000","last_undegraded":"2024-09-14T05:44:35.197257+0000","last_fullsized":"2024-09-14T05:44:35.197257+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3789","last_scrub_stamp":"2024-09-14T05:44:35.196981+0000","last_deep_scrub":"37'3789","last_deep_scrub_stamp":"2024-09-14T05:44:35.196981+0000","last_clean_scrub_stamp":"2024-09-14T05:44:35.196981+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:42.175827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00087165300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":282428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"37'4085","reported_seq":6600,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:36.362531+0000","last_change":"2024-09-14T05:44:36.362531+0000","last_active":"2024-09-14T05:44:36.362531+0000","last_peered":"2024-09-14T05:44:36.362531+0000","last_clean":"2024-09-14T05:44:36.362531+0000","last_became_active":"2024-09-14T05:13:34.572679+0000","last_became_peered":"2024-09-14T05:13:34.572679+0000","last_unstale":"2024-09-14T05:44:36.362531+0000","last_undegraded":"2024-09-14T05:44:36.362531+0000","last_fullsized":"2024-09-14T05:44:36.362531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'4085","last_scrub_stamp":"2024-09-14T05:44:36.362385+0000","last_deep_scrub":"37'4085","last_deep_scrub_stamp":"2024-09-14T05:44:36.362385+0000","last_clean_scrub_stamp":"2024-09-14T05:44:36.362385+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:15.440992+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00095268499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5775,"num_read_kb":5061,"num_write":4018,"num_write_kb":312778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"37'3987","reported_seq":4076,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:35.384267+0000","last_change":"2024-09-14T05:44:35.384010+0000","last_active":"2024-09-14T05:44:35.384267+0000","last_peered":"2024-09-14T05:44:35.384267+0000","last_clean":"2024-09-14T05:44:35.384267+0000","last_became_active":"2024-09-14T05:13:34.569117+0000","last_became_peered":"2024-09-14T05:13:34.569117+0000","last_unstale":"2024-09-14T05:44:35.384267+0000","last_undegraded":"2024-09-14T05:44:35.384267+0000","last_fullsized":"2024-09-14T05:44:35.384267+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'3987","last_scrub_stamp":"2024-09-14T05:44:35.383742+0000","last_deep_scrub":"37'3987","last_deep_scrub_stamp":"2024-09-14T05:44:35.383742+0000","last_clean_scrub_stamp":"2024-09-14T05:44:35.383742+0000","objects_scrubbed":1,"log_size":3987,"log_dups_size":0,"ondisk_log_size":3987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:32.108550+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015874839999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":3986,"num_write_kb":308587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"37'74","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-14T05:44:34.396897+0000","last_change":"2024-09-14T05:44:34.396897+0000","last_active":"2024-09-14T05:44:34.396897+0000","last_peered":"2024-09-14T05:44:34.396897+0000","last_clean":"2024-09-14T05:44:34.396897+0000","last_became_active":"2024-09-14T05:13:27.696024+0000","last_became_peered":"2024-09-14T05:13:27.696024+0000","last_unstale":"2024-09-14T05:44:34.396897+0000","last_undegraded":"2024-09-14T05:44:34.396897+0000","last_fullsized":"2024-09-14T05:44:34.396897+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'74","last_scrub_stamp":"2024-09-14T05:44:34.396739+0000","last_deep_scrub":"37'74","last_deep_scrub_stamp":"2024-09-14T05:44:34.396739+0000","last_clean_scrub_stamp":"2024-09-14T05:44:34.396739+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:56:12.734167+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21771,"num_read_kb":18348,"num_write":36086,"num_write_kb":2542978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33613,"ondisk_log_size":33613,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949673340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":50140,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":104807460,"statfs":{"total":107374182400,"available":107322839040,"internally_reserved":0,"allocated":929792,"data_stored":722879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949673340,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":89816,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":104767784,"statfs":{"total":107374182400,"available":107282210816,"internally_reserved":0,"allocated":335872,"data_stored":132511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":53716,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":104803884,"statfs":{"total":107374182400,"available":107319177216,"internally_reserved":0,"allocated":331776,"data_stored":132492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706044,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":90392,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":104767208,"statfs":{"total":107374182400,"available":107281620992,"internally_reserved":0,"allocated":925696,"data_stored":722860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:44:54.426 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-14T05:44:54.742 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T05:44:54.742 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T05:44:54.742 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-14T05:44:54.742 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:44:54.814 INFO:tasks.ceph.osd.0:Stopped 2024-09-14T05:44:54.815 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-14T05:44:54.815 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:44:54.908 INFO:tasks.ceph.osd.1:Stopped 2024-09-14T05:44:54.908 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-14T05:44:54.908 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:44:54.986 INFO:tasks.ceph.osd.2:Stopped 2024-09-14T05:44:54.986 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-14T05:44:54.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:44:58.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-14T05:45:01.088 INFO:tasks.ceph.osd.3:Stopped 2024-09-14T05:45:01.089 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T05:45:01.089 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-14T05:45:01.089 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:45:01.120 INFO:tasks.ceph.mgr.x:Stopped 2024-09-14T05:45:01.120 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-14T05:45:01.120 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:45:01.148 INFO:tasks.ceph.mgr.y:Stopped 2024-09-14T05:45:01.149 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T05:45:01.149 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-14T05:45:01.149 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:45:01.176 INFO:tasks.ceph.mon.a:Stopped 2024-09-14T05:45:01.176 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-14T05:45:01.176 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T05:45:01.201 INFO:tasks.ceph.mon.b:Stopped 2024-09-14T05:45:01.202 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T05:45:01.202 DEBUG:teuthology.orchestra.run.smithi144:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-14T05:45:01.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:45:01.237 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-14T05:45:01.484 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:45:01.484 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-14T05:45:01.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:45:01.646 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-14T05:45:01.836 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:45:01.837 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-14T05:45:02.012 INFO:tasks.ceph:Archiving mon data... 2024-09-14T05:45:02.014 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/data/mon.a.tgz 2024-09-14T05:45:02.014 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-14T05:45:02.020 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.w31ckpmrHM 2024-09-14T05:45:02.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.w31ckpmrHM 2024-09-14T05:45:02.131 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.w31ckpmrHM 2024-09-14T05:45:02.196 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.w31ckpmrHM is 267KB 2024-09-14T05:45:02.257 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.w31ckpmrHM 2024-09-14T05:45:02.262 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/data/mon.b.tgz 2024-09-14T05:45:02.262 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-14T05:45:02.269 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.aDcXsECfcM 2024-09-14T05:45:02.269 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.aDcXsECfcM 2024-09-14T05:45:02.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.aDcXsECfcM 2024-09-14T05:45:02.441 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.aDcXsECfcM is 275KB 2024-09-14T05:45:02.540 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.aDcXsECfcM 2024-09-14T05:45:02.546 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T05:45:02.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T05:45:02.548 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T05:45:02.609 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T05:45:02.650 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T05:45:02.650 INFO:tasks.ceph:Archiving crash dumps... 2024-09-14T05:45:02.652 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi144/crash 2024-09-14T05:45:02.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T05:45:02.667 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi194/crash 2024-09-14T05:45:02.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T05:45:02.706 INFO:tasks.ceph:Compressing logs... 2024-09-14T05:45:02.706 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T05:45:02.712 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T05:45:02.734 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26950.log 2024-09-14T05:45:02.735 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26496.log 2024-09-14T05:45:02.735 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26820.log 2024-09-14T05:45:02.735 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26950.log.gz 2024-09-14T05:45:02.735 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26399.log 2024-09-14T05:45:02.735 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26496.log.gz 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26820.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26820.log.gz 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26399.log.gz 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27002.log 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26195.log 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.27002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27002.log.gz 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26768.log 2024-09-14T05:45:02.742 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26794.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26195.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26768.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23310.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26392.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26794.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.23310.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.25931.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.23310.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26296.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26392.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.25931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26976.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25931.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26296.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26296.log.gz 2024-09-14T05:45:02.743 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25906.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26976.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26664.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.25906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25906.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25976.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26522.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26664.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.25976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25976.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26196.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26522.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.26522.log.gz -- 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr: /var/log/ceph/ceph-client.admin.26184.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26196.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26924.log 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26184.log.gz 2024-09-14T05:45:02.744 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26690.log 2024-09-14T05:45:02.753 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26924.log.gz 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26054.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26548.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26690.log.gz 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26054.log.gz 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26846.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26548.log.gz 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.26846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26846.log.gz 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26300.log 2024-09-14T05:45:02.754 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26288.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.26300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25851.log 0.0% 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26300.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26716.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26288.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.25851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26898.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25851.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23211.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26716.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26092.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26898.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.23211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26089.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23211.log.gz 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26092.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-14T05:45:02.755 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26092.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: 88.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27026.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26089.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26028.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.27026.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26400.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27026.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26188.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26028.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26405.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26400.log.gz 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26294.log 2024-09-14T05:45:02.756 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26188.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:gzip/var/log/ceph/ceph-client.admin.26405.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21522.log 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26405.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26742.log 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26294.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26087.log 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.tmp-client.admin.21522.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21522.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25956.log 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26742.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:gzip/var/log/ceph/ceph-client.admin.26087.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26091.log 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26087.log.gz 2024-09-14T05:45:02.757 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.25956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23360.log 2024-09-14T05:45:02.758 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25956.log.gz 2024-09-14T05:45:02.758 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26872.log 2024-09-14T05:45:02.758 INFO:teuthology.orchestra.run.smithi144.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26091.log.gz 2024-09-14T05:45:02.758 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.23360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23360.log.gz 2024-09-14T05:45:02.758 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26002.log 2024-09-14T05:45:02.759 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26872.log.gz 2024-09-14T05:45:02.759 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.26002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26002.log.gz 2024-09-14T05:45:02.771 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26245.log 2024-09-14T05:45:02.772 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26386.log 2024-09-14T05:45:02.773 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26854.log 2024-09-14T05:45:02.773 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26245.log.gz 2024-09-14T05:45:02.773 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26386.log.gz 2024-09-14T05:45:02.773 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25750.log 2024-09-14T05:45:02.774 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27007.log 2024-09-14T05:45:02.774 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26854.log.gz 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23275.log 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25750.log.gz 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.27007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27007.log.gz 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.23275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23275.log.gz 2024-09-14T05:45:02.775 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26829.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:gzip/var/log/ceph/ceph-mgr.y.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26985.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26829.log.gz 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26552.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23251.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26985.log.gz 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26552.log.gz 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25893.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26669.log 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.23251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23251.log.gz 2024-09-14T05:45:02.777 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25893.log.gz 2024-09-14T05:45:02.778 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26199.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26082.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26669.log: /var/log/ceph/ceph-client.admin.26199.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.26669.log.gz 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr: --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26199.log.gz 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25657.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26082.log.gz 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25819.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26786.log 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25657.log.gz 2024-09-14T05:45:02.780 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25819.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.26334.log 2024-09-14T05:45:02.781 INFO:teuthology.orchestra.run.smithi194.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25819.log.gz 2024-09-14T05:45:02.781 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26786.log.gz 2024-09-14T05:45:02.781 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26131.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26808.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26334.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26131.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23299.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5/var/log/ceph/ceph-client.admin.26808.log: --verbose -- /var/log/ceph/ceph-client.admin.26717.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26808.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.23299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23299.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26432.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.26717.log: /var/log/ceph/ceph-client.admin.26478.log 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26717.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26432.log.gz 2024-09-14T05:45:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25796.log 2024-09-14T05:45:02.785 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26601.log 2024-09-14T05:45:02.785 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26478.log.gz 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25796.log.gz 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25771.log 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26601.log.gz 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25771.log.gz 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25680.log 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:gzip/var/log/ceph/ceph-osd.3.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26313.log 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25680.log.gz 2024-09-14T05:45:02.786 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26360.log 2024-09-14T05:45:02.788 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26690.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26313.log: /var/log/ceph/ceph-client.admin.26360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26313.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26360.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26963.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26457.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26690.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26224.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26963.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26457.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26531.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26224.log: /var/log/ceph/ceph-client.admin.26531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26224.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26531.log.gz 2024-09-14T05:45:02.789 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25587.log 2024-09-14T05:45:02.790 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26740.log 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.25587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26291.log 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25587.log.gz 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26740.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.26740.log.gz -5 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr: --verbose -- /var/log/ceph/ceph-client.admin.26031.log 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25608.log 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26291.log.gz 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26031.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26031.log.gz 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr: --verbose -- /var/log/ceph/ceph-client.admin.25562.log 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25608.log.gz 2024-09-14T05:45:02.795 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25701.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25562.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25562.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.26178.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25701.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23323.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26178.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26648.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.23323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26009.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23323.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr: 88.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26648.log.gz 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26622.log 2024-09-14T05:45:02.796 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25984.log 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26009.log.gz 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26622.log.gz 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26761.log 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25984.log: /var/log/ceph/ceph.audit.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25984.log.gz 2024-09-14T05:45:02.797 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25941.log 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26503.log 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26761.log.gz 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25963.log 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25941.log.gz 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26503.log.gz 2024-09-14T05:45:02.803 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26938.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26110.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25963.log: /var/log/ceph/ceph-client.admin.26938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25963.log.gz 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26938.log.gz 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25544.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26156.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26110.log.gz 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25914.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25544.log.gz 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26156.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.26156.log.gz -- 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr: /var/log/ceph/ceph-client.admin.25840.log 2024-09-14T05:45:02.804 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25914.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25914.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25868.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25840.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26266.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25726.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25868.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26917.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26266.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26579.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25726.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26052.log 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26917.log.gz 2024-09-14T05:45:02.805 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26407.log 2024-09-14T05:45:02.806 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26579.log.gz 2024-09-14T05:45:02.806 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25629.log 2024-09-14T05:45:02.806 INFO:teuthology.orchestra.run.smithi194.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26052.log.gz 2024-09-14T05:45:02.806 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.26407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26407.log.gz 2024-09-14T05:45:02.806 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.25629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25629.log.gz 2024-09-14T05:45:02.922 INFO:teuthology.orchestra.run.smithi144.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-14T05:45:03.170 INFO:teuthology.orchestra.run.smithi194.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-14T05:45:03.335 INFO:teuthology.orchestra.run.smithi144.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-14T05:45:03.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-14T05:45:03.599 INFO:teuthology.orchestra.run.smithi194.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-14T05:45:03.765 INFO:teuthology.orchestra.run.smithi144.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-14T05:45:06.679 INFO:teuthology.orchestra.run.smithi144.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-14T05:45:06.681 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-14T05:45:06.681 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m3.967s 2024-09-14T05:45:06.681 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m5.572s 2024-09-14T05:45:06.681 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m0.243s 2024-09-14T05:45:07.101 INFO:teuthology.orchestra.run.smithi194.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-14T05:45:07.102 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-09-14T05:45:07.102 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m4.348s 2024-09-14T05:45:07.102 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m5.427s 2024-09-14T05:45:07.102 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.204s 2024-09-14T05:45:07.103 INFO:tasks.ceph:Archiving logs... 2024-09-14T05:45:07.103 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi144/log 2024-09-14T05:45:07.104 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T05:45:07.874 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi194/log 2024-09-14T05:45:07.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T05:45:08.655 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-14T05:45:08.665 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-14T05:45:08.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T05:45:08.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T05:45:08.732 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-14T05:45:08.741 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-14T05:45:08.741 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:45:08.743 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:45:08.752 INFO:teuthology.orchestra.run.smithi144.stdout: remote refid st t when poll reach delay offset jitter 2024-09-14T05:45:08.753 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================== 2024-09-14T05:45:08.753 INFO:teuthology.orchestra.run.smithi144.stdout:+hv01.front.sepi 67.205.162.81 3 u 14 64 377 0.077 -3.685 1.407 2024-09-14T05:45:08.753 INFO:teuthology.orchestra.run.smithi144.stdout:*hv02.front.sepi 63.231.80.2 3 u 67 64 377 0.111 -2.354 1.045 2024-09-14T05:45:08.753 INFO:teuthology.orchestra.run.smithi144.stdout:+hv03.front.sepi 74.6.168.72 3 u 12 64 377 0.093 -1.580 1.218 2024-09-14T05:45:08.753 INFO:teuthology.orchestra.run.smithi144.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: remote refid st t when poll reach delay offset jitter 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================== 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:*hv01.front.sepi 67.205.162.81 3 u 67 64 377 0.107 -2.319 1.128 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.094 -3.543 0.864 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:+hv03.front.sepi 74.6.168.72 3 u 56 64 377 0.116 -2.443 0.953 2024-09-14T05:45:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-14T05:45:08.756 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-14T05:45:08.765 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-14T05:45:08.765 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-14T05:45:08.775 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-14T05:45:08.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-14T05:45:08.793 INFO:teuthology.task.internal:Duration was 2493.239847 seconds 2024-09-14T05:45:08.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-14T05:45:08.802 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-14T05:45:08.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T05:45:08.805 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T05:45:08.842 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-14T05:45:08.842 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-09-14T05:45:08.842 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T05:45:08.891 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-09-14T05:45:08.891 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T05:45:08.901 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-14T05:45:08.901 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T05:45:08.936 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T05:45:08.972 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-14T05:45:08.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T05:45:09.004 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T05:45:09.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-14T05:45:09.084 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-14T05:45:09.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T05:45:09.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T05:45:09.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-14T05:45:09.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:45:09.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:45:09.149 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-09-14T05:45:09.162 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-09-14T05:45:09.227 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:45:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:45:09.275 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:45:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:45:09.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-14T05:45:09.289 INFO:teuthology.task.internal:Transferring archived files... 2024-09-14T05:45:09.290 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi144 2024-09-14T05:45:09.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T05:45:09.341 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900293/remote/smithi194 2024-09-14T05:45:09.342 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T05:45:09.363 INFO:teuthology.task.internal:Removing archive directory... 2024-09-14T05:45:09.363 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T05:45:09.381 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T05:45:09.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-14T05:45:09.419 INFO:teuthology.task.internal:Not uploading archives. 2024-09-14T05:45:09.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-14T05:45:09.429 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-14T05:45:09.429 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T05:45:09.432 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T05:45:09.435 INFO:teuthology.orchestra.run.smithi144.stdout: 658615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 05:45 /home/ubuntu/cephtest 2024-09-14T05:45:09.458 INFO:teuthology.orchestra.run.smithi194.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 05:45 /home/ubuntu/cephtest 2024-09-14T05:45:09.459 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-14T05:45:09.469 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-14T05:45:09.492 INFO:teuthology.run:Summary data: description: rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/recovery conf/{disable-pool-app}} duration: 2493.2398467063904 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-14T05:45:09.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T05:45:09.565 INFO:teuthology.run:pass