2024-05-08T15:46:54.324 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-08T15:46:54.324 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_lflores@teuthology --description rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} --name lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631 -- /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631/orig.config.yaml 2024-05-08T15:46:54.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T15:46:54.447 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631 branch: wip-lflores-testing-2-2024-05-07-1606-squid description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} email: null first_in_suite: false job_id: '7697631' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lflores-testing-2-2024-05-07-1606-squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 workunit: branch: wip-lflores-testing-2-2024-05-07-1606-squid sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 seed: 8082 sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-lflores-testing-2-2024-05-07-1606-squid suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjJIzyZqhpoq7p/b5uN9kUUI/hwkd8ztS5LtUwbkEruEs+8X7SfR84X0yyKHIBtz2V1O1ILR45DfCoMfmcPudo= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_14:59:36 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-08T15:46:54.447 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa; will attempt to use it 2024-05-08T15:46:54.448 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa/tasks 2024-05-08T15:46:54.448 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-08T15:46:54.449 INFO:teuthology.task.internal:Checking packages... 2024-05-08T15:46:54.477 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '95a4ad3c90f2f12f920e87725ad8d9002f948722' 2024-05-08T15:46:54.477 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-08T15:46:54.477 INFO:teuthology.packaging:ref: None 2024-05-08T15:46:54.477 INFO:teuthology.packaging:tag: None 2024-05-08T15:46:54.478 INFO:teuthology.packaging:branch: wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T15:46:54.478 INFO:teuthology.packaging:sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:46:54.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T15:46:54.745 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:46:54.746 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-08T15:46:54.753 INFO:teuthology.task.internal:no buildpackages task found 2024-05-08T15:46:54.753 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-08T15:46:54.760 INFO:teuthology.task.internal:Saving configuration 2024-05-08T15:46:54.771 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-08T15:46:54.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-08T15:46:54.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 15:41:25.233977', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE62Nf8OrTDjuT5zBOjSoSu4s7uv5rcySHdff93rg5F3'} 2024-05-08T15:46:54.802 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-08T15:46:54.807 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2024-05-08T15:46:54.807 INFO:teuthology.run_tasks:Running task console_log... 2024-05-08T15:46:54.849 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe9fb1b1e50>, signals=[15]) 2024-05-08T15:46:54.850 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-08T15:46:54.857 INFO:teuthology.task.internal:Opening connections... 2024-05-08T15:46:54.857 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:46:54.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:46:54.955 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-08T15:46:54.962 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-05-08T15:46:55.055 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-05-08T15:46:55.055 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Ubuntu" 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="22.04" 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_CODENAME=jammy 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:ID=ubuntu 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE=debian 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-08T15:46:55.104 INFO:teuthology.orchestra.run.smithi002.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-08T15:46:55.105 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-08T15:46:55.105 INFO:teuthology.orchestra.run.smithi002.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-08T15:46:55.105 INFO:teuthology.orchestra.run.smithi002.stdout:UBUNTU_CODENAME=jammy 2024-05-08T15:46:55.105 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-05-08T15:46:55.134 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-08T15:46:55.143 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-08T15:46:55.149 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-08T15:46:55.149 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-05-08T15:46:55.154 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-08T15:46:55.160 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-08T15:46:55.160 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-05-08T15:46:55.202 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-08T15:46:55.235 INFO:teuthology.run_tasks:Running task kernel... 2024-05-08T15:46:55.250 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-08T15:46:55.251 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-08T15:46:55.251 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T15:46:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:46:55.256 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-05-08T15:46:55.303 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-56-generic 2024-05-08T15:46:55.303 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.15.0-56-generic 2024-05-08T15:46:55.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2024-05-08T15:46:55.426 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-05-08T15:46:55.564 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-08T15:46:55.674 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:46:55.755 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-08T15:46:55.804 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [458 kB] 2024-05-08T15:46:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,421 kB] 2024-05-08T15:46:56.002 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [246 kB] 2024-05-08T15:46:56.007 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-08T15:46:56.008 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB] 2024-05-08T15:46:56.066 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-08T15:46:56.067 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB] 2024-05-08T15:46:56.083 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-08T15:46:56.083 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [602 kB] 2024-05-08T15:46:56.109 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [849 kB] 2024-05-08T15:46:56.116 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-08T15:46:56.119 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-05-08T15:46:56.120 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-08T15:46:56.121 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-08T15:46:56.121 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-08T15:46:56.145 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-08T15:46:56.145 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-08T15:46:56.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB] 2024-05-08T15:46:56.568 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB] 2024-05-08T15:46:56.608 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB] 2024-05-08T15:46:56.623 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-08T15:46:56.623 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-08T15:46:56.625 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB] 2024-05-08T15:46:56.686 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB] 2024-05-08T15:46:56.693 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-08T15:46:56.858 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-08T15:46:57.445 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,073 kB] 2024-05-08T15:46:57.555 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-08T15:46:57.576 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-08T15:46:57.577 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-08T15:46:57.581 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-08T15:46:57.581 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-08T15:46:57.582 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-08T15:46:57.582 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-08T15:46:57.586 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-08T15:46:57.590 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-08T15:46:57.755 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-08T15:46:57.837 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-08T15:46:57.947 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-08T15:46:58.001 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-08T15:46:58.031 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-08T15:47:02.598 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 13.4 MB in 3s (5,272 kB/s) 2024-05-08T15:47:03.993 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:47:04.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-08T15:47:04.075 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:47:04.277 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-05-08T15:47:04.278 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-05-08T15:47:04.461 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:47:04.462 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:47:04.462 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:47:04.462 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:47:04.462 INFO:teuthology.orchestra.run.smithi002.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-08T15:47:04.463 INFO:teuthology.orchestra.run.smithi002.stdout: python2.7-minimal 2024-05-08T15:47:04.463 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:47:04.463 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2024-05-08T15:47:04.464 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-08T15:47:04.464 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-generic linux-image-5.15.0-106-generic 2024-05-08T15:47:04.464 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic 2024-05-08T15:47:04.465 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2024-05-08T15:47:04.465 INFO:teuthology.orchestra.run.smithi002.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-08T15:47:04.466 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2024-05-08T15:47:04.466 INFO:teuthology.orchestra.run.smithi002.stdout: thermald 2024-05-08T15:47:04.516 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2024-05-08T15:47:04.516 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-08T15:47:04.516 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic 2024-05-08T15:47:04.516 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-extra-5.15.0-106-generic 2024-05-08T15:47:04.518 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages will be upgraded: 2024-05-08T15:47:04.518 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-08T15:47:04.578 INFO:teuthology.orchestra.run.smithi002.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-05-08T15:47:04.578 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 113 MB of archives. 2024-05-08T15:47:04.578 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-08T15:47:04.578 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB] 2024-05-08T15:47:05.144 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB] 2024-05-08T15:47:05.439 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB] 2024-05-08T15:47:06.656 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B] 2024-05-08T15:47:06.656 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B] 2024-05-08T15:47:06.657 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB] 2024-05-08T15:47:07.082 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB] 2024-05-08T15:47:07.190 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B] 2024-05-08T15:47:07.609 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 113 MB in 3s (42.6 MB/s) 2024-05-08T15:47:07.790 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic. 2024-05-08T15:47:09.798 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-08T15:47:09.802 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:47:09.988 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:12.189 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic. 2024-05-08T15:47:12.210 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:47:12.295 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:12.758 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic. 2024-05-08T15:47:12.779 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:47:12.816 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:19.693 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:47:19.819 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:47:20.136 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:47:20.248 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:47:20.509 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-106. 2024-05-08T15:47:20.520 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ... 2024-05-08T15:47:20.559 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-08T15:47:26.332 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic. 2024-05-08T15:47:26.369 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:47:26.398 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:29.030 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:47:29.136 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:47:29.469 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-08T15:47:29.590 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:29.708 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-generic (5.15.0.106.106) ... 2024-05-08T15:47:29.837 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:31.239 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:47:31.239 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-08T15:47:31.239 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:47:31.239 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic 2024-05-08T15:47:31.989 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:33.432 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:34.867 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-generic (5.15.0.106.106) ... 2024-05-08T15:47:34.993 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-generic (5.15.0.106.106) ... 2024-05-08T15:47:35.119 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:47:35.178 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-08T15:47:35.178 INFO:teuthology.orchestra.run.smithi002.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:47:49.590 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-08T15:47:49.591 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub' 2024-05-08T15:47:49.603 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-08T15:47:49.628 INFO:teuthology.orchestra.run.smithi002.stdout:Generating grub configuration file ... 2024-05-08T15:47:49.885 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:47:49.904 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:47:50.196 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:47:50.201 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-08T15:47:50.273 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-08T15:47:50.279 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-08T15:47:50.458 INFO:teuthology.orchestra.run.smithi002.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-08T15:47:50.458 INFO:teuthology.orchestra.run.smithi002.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-08T15:47:50.458 INFO:teuthology.orchestra.run.smithi002.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-08T15:47:50.492 INFO:teuthology.orchestra.run.smithi002.stdout:done 2024-05-08T15:47:50.936 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2024-05-08T15:47:50.952 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 21 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.106.106 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2024-05-08T15:47:50.953 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2024-05-08T15:47:50.954 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2024-05-08T15:47:50.954 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic 2024-05-08T15:47:50.954 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T15:47:50.954 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T15:47:50.954 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-08T15:47:50.954 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T15:47:50.954 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2024-05-08T15:47:51.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-05-08T15:47:51.250 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:47:51.261 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-08T15:47:51.281 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-08T15:47:51.366 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-08T15:47:52.666 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:47:52.684 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-08T15:47:52.747 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:47:52.952 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-05-08T15:47:52.953 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-05-08T15:47:53.134 INFO:teuthology.orchestra.run.smithi002.stdout:linux-image-generic is already the newest version (5.15.0.106.106). 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-08T15:47:53.135 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-08T15:47:53.136 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:47:53.177 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-05-08T15:47:53.178 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2024-05-08T15:47:53.194 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 21 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.106.106 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2024-05-08T15:47:53.195 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2024-05-08T15:47:53.196 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-05-08T15:47:53.242 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.vyfdKRjZFw 2024-05-08T15:47:53.242 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vyfdKRjZFw 2024-05-08T15:47:53.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.vyfdKRjZFw 2024-05-08T15:47:53.415 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.vyfdKRjZFw is 10KB 2024-05-08T15:47:53.429 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.vyfdKRjZFw 2024-05-08T15:47:53.434 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-08T15:47:53.490 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:47:53.490 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-08T15:47:53.490 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-08T15:47:53.558 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic 2024-05-08T15:47:53.558 DEBUG:teuthology.orchestra.run.smithi002:> sudo update-grub 2024-05-08T15:47:54.737 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub' 2024-05-08T15:47:54.738 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-08T15:47:54.740 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-05-08T15:47:54.921 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:47:54.927 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:47:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:47:55.162 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-08T15:47:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-08T15:47:55.210 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-08T15:47:55.359 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-08T15:47:55.359 INFO:teuthology.orchestra.run.smithi002.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-08T15:47:55.359 INFO:teuthology.orchestra.run.smithi002.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-08T15:47:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-05-08T15:47:55.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-05-08T15:48:25.423 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-08T15:48:25.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:48:25.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:48:43.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2024-05-08T15:48:52.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:48:52.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:48:56.002 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2024-05-08T15:49:08.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:49:08.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:49:39.888 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-05-08T15:49:40.601 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:49:40.601 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-106-generic"... 2024-05-08T15:49:40.601 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-05-08T15:49:40.648 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-106-generic 2024-05-08T15:49:40.648 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic 2024-05-08T15:49:40.649 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T15:49:40.649 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-08T15:49:41.649 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-08T15:49:41.649 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T15:49:41.744 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-05-08T15:49:41.789 DEBUG:teuthology.parallel:result is None 2024-05-08T15:49:41.789 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-08T15:49:41.800 INFO:teuthology.task.internal:Creating test directory... 2024-05-08T15:49:41.800 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T15:49:41.806 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-08T15:49:41.813 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-08T15:49:41.867 INFO:teuthology.task.internal:Creating archive directory... 2024-05-08T15:49:41.867 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T15:49:41.889 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-08T15:49:41.896 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-08T15:49:41.896 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T15:49:41.956 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:49:41.963 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:49:41.964 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-08T15:49:41.971 INFO:teuthology.task.internal:Configuring sudo... 2024-05-08T15:49:41.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T15:49:42.020 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-08T15:49:42.029 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-08T15:49:42.029 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T15:49:42.067 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T15:49:42.119 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T15:49:42.168 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:49:42.168 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T15:49:42.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-05-08T15:49:42.311 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-08T15:49:42.319 INFO:teuthology.task.internal:Starting timer... 2024-05-08T15:49:42.319 INFO:teuthology.run_tasks:Running task pcp... 2024-05-08T15:49:42.330 INFO:teuthology.run_tasks:Running task selinux... 2024-05-08T15:49:42.339 DEBUG:teuthology.task.selinux:Excluding smithi002: OS 'ubuntu' does not support SELinux 2024-05-08T15:49:42.340 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-08T15:49:42.340 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-08T15:49:42.340 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-08T15:49:42.340 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-08T15:49:42.350 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-08T15:49:42.368 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-08T15:49:42.369 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-08T15:56:52.402 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com')] 2024-05-08T15:56:52.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:56:52.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:56:52.491 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-05-08T15:56:52.575 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-05-08T15:56:52.575 INFO:teuthology.run_tasks:Running task clock... 2024-05-08T15:56:52.586 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-08T15:56:52.587 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T15:56:52.587 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T15:56:52.646 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Command line: ntpd -gq 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: ---------------------------------------------------- 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: ntp-4 is maintained by Network Time Foundation, 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: corporation. Support and training for ntp-4 are 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: available at https://www.nwtime.org/support 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: ---------------------------------------------------- 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: proto: precision = 0.046 usec (-24) 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: basedate set to 2022-02-04 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: gps base set to 2022-02-06 (week 2196) 2024-05-08T15:56:52.647 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-08T15:56:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-08T15:56:52.648 INFO:teuthology.orchestra.run.smithi002.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: restrict ::: KOD does nothing without LIMITED. 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen and drop on 0 v6wildcard [::]:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen normally on 2 lo 127.0.0.1:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen normally on 3 ens1f0 172.21.15.2:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen normally on 4 lo [::1]:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1920%4]:123 2024-05-08T15:56:52.649 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:52 ntpd[17142]: Listening on routing socket on fd #22 for interface updates 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:53.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:53 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:55.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:55 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:57.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:57.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:57 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:59.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:59.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:59.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:59.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:59.647 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stdout: 8 May 15:56:59 ntpd[17142]: ntpd: time slew -0.002789 s 2024-05-08T15:56:59.648 INFO:teuthology.orchestra.run.smithi002.stdout:ntpd: time slew -0.002789s 2024-05-08T15:56:59.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-08T15:56:59.649 INFO:teuthology.orchestra.run.smithi002.stderr: 8 May 15:56:59 ntpd[17142]: can't open /var/log/ntpstats/loopstats.20240508: Permission denied 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:56:59.694 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:56:59.695 INFO:teuthology.run_tasks:Running task install... 2024-05-08T15:56:59.705 DEBUG:teuthology.task.install:project ceph 2024-05-08T15:56:59.705 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'}} 2024-05-08T15:56:59.705 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'} 2024-05-08T15:56:59.706 INFO:teuthology.task.install:Using flavor: default 2024-05-08T15:56:59.711 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-08T15:56:59.711 INFO:teuthology.task.install:extra packages: [] 2024-05-08T15:56:59.712 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-key list | grep Ceph 2024-05-08T15:56:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-08T15:56:59.825 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-08T15:56:59.826 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph.com (release key) 2024-05-08T15:56:59.826 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-08T15:56:59.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:56:59.978 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default/ 2024-05-08T15:56:59.978 INFO:teuthology.task.install.deb:Package version is 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:57:00.095 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:57:00.095 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-08T15:57:00.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2024-05-08T15:57:00.286 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:57:00.286 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-08T15:57:00.303 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-08T15:57:00.326 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-08T15:57:00.416 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy InRelease 2024-05-08T15:57:00.454 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-08T15:57:00.499 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-08T15:57:00.537 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-05-08T15:57:00.581 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-05-08T15:57:00.662 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 28.9 kB in 0s (77.1 kB/s) 2024-05-08T15:57:01.647 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:57:01.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2517-g95a4ad3c-1jammy cephadm=19.0.0-2517-g95a4ad3c-1jammy ceph-mds=19.0.0-2517-g95a4ad3c-1jammy ceph-mgr=19.0.0-2517-g95a4ad3c-1jammy ceph-common=19.0.0-2517-g95a4ad3c-1jammy ceph-fuse=19.0.0-2517-g95a4ad3c-1jammy ceph-test=19.0.0-2517-g95a4ad3c-1jammy ceph-volume=19.0.0-2517-g95a4ad3c-1jammy radosgw=19.0.0-2517-g95a4ad3c-1jammy python3-rados=19.0.0-2517-g95a4ad3c-1jammy python3-rgw=19.0.0-2517-g95a4ad3c-1jammy python3-cephfs=19.0.0-2517-g95a4ad3c-1jammy python3-rbd=19.0.0-2517-g95a4ad3c-1jammy libcephfs2=19.0.0-2517-g95a4ad3c-1jammy libcephfs-dev=19.0.0-2517-g95a4ad3c-1jammy librados2=19.0.0-2517-g95a4ad3c-1jammy librbd1=19.0.0-2517-g95a4ad3c-1jammy rbd-fuse=19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:57:01.727 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2024-05-08T15:57:01.932 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2024-05-08T15:57:01.932 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2024-05-08T15:57:02.136 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:57:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:57:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:57:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:57:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-08T15:57:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-08T15:57:02.137 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:57:02.137 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2024-05-08T15:57:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-08T15:57:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-08T15:57:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-08T15:57:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet zip 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-08T15:57:02.141 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2024-05-08T15:57:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: btrfs-tools 2024-05-08T15:57:02.248 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2024-05-08T15:57:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-08T15:57:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-08T15:57:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-08T15:57:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-08T15:57:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-08T15:57:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-08T15:57:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-08T15:57:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-05-08T15:57:02.365 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 118 newly installed, 0 to remove and 303 not upgraded. 2024-05-08T15:57:02.365 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 180 MB of archives. 2024-05-08T15:57:02.365 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 822 MB of additional disk space will be used. 2024-05-08T15:57:02.365 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-08T15:57:02.582 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2517-g95a4ad3c-1jammy [3,546 kB] 2024-05-08T15:57:02.600 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-08T15:57:02.606 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-08T15:57:02.653 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-08T15:57:02.856 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-08T15:57:02.862 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-08T15:57:02.879 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-08T15:57:02.885 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-08T15:57:02.886 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-08T15:57:02.887 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-08T15:57:02.900 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-08T15:57:02.903 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-08T15:57:02.906 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-08T15:57:02.909 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-08T15:57:02.965 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-08T15:57:02.970 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-08T15:57:02.974 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-08T15:57:03.019 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-08T15:57:03.020 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-08T15:57:03.052 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-08T15:57:03.055 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-08T15:57:03.055 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-08T15:57:03.056 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-08T15:57:03.057 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-08T15:57:03.058 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-08T15:57:03.058 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-08T15:57:03.076 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-08T15:57:03.082 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-08T15:57:03.101 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-08T15:57:03.102 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-08T15:57:03.103 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-08T15:57:03.105 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-08T15:57:03.105 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-08T15:57:03.106 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-08T15:57:03.109 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-08T15:57:03.110 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-08T15:57:03.132 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-08T15:57:03.143 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-08T15:57:03.163 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-08T15:57:03.164 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-08T15:57:03.165 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-08T15:57:03.168 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-08T15:57:03.172 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-08T15:57:03.173 INFO:teuthology.orchestra.run.smithi002.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-08T15:57:03.176 INFO:teuthology.orchestra.run.smithi002.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-08T15:57:03.178 INFO:teuthology.orchestra.run.smithi002.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-08T15:57:03.192 INFO:teuthology.orchestra.run.smithi002.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-08T15:57:03.194 INFO:teuthology.orchestra.run.smithi002.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-08T15:57:03.224 INFO:teuthology.orchestra.run.smithi002.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-08T15:57:03.228 INFO:teuthology.orchestra.run.smithi002.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-08T15:57:03.388 INFO:teuthology.orchestra.run.smithi002.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-08T15:57:03.390 INFO:teuthology.orchestra.run.smithi002.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-08T15:57:03.394 INFO:teuthology.orchestra.run.smithi002.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-08T15:57:03.394 INFO:teuthology.orchestra.run.smithi002.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-08T15:57:03.397 INFO:teuthology.orchestra.run.smithi002.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-08T15:57:03.432 INFO:teuthology.orchestra.run.smithi002.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-08T15:57:03.435 INFO:teuthology.orchestra.run.smithi002.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-08T15:57:03.435 INFO:teuthology.orchestra.run.smithi002.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-08T15:57:03.457 INFO:teuthology.orchestra.run.smithi002.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-08T15:57:03.457 INFO:teuthology.orchestra.run.smithi002.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-08T15:57:03.475 INFO:teuthology.orchestra.run.smithi002.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-08T15:57:03.475 INFO:teuthology.orchestra.run.smithi002.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-08T15:57:03.476 INFO:teuthology.orchestra.run.smithi002.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-08T15:57:03.476 INFO:teuthology.orchestra.run.smithi002.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-08T15:57:03.477 INFO:teuthology.orchestra.run.smithi002.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-08T15:57:03.477 INFO:teuthology.orchestra.run.smithi002.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-08T15:57:03.481 INFO:teuthology.orchestra.run.smithi002.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-08T15:57:03.496 INFO:teuthology.orchestra.run.smithi002.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-08T15:57:03.497 INFO:teuthology.orchestra.run.smithi002.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-08T15:57:03.498 INFO:teuthology.orchestra.run.smithi002.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-08T15:57:03.516 INFO:teuthology.orchestra.run.smithi002.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-08T15:57:03.520 INFO:teuthology.orchestra.run.smithi002.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-08T15:57:03.521 INFO:teuthology.orchestra.run.smithi002.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-08T15:57:03.521 INFO:teuthology.orchestra.run.smithi002.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-08T15:57:03.527 INFO:teuthology.orchestra.run.smithi002.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-08T15:57:03.527 INFO:teuthology.orchestra.run.smithi002.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-08T15:57:03.532 INFO:teuthology.orchestra.run.smithi002.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-08T15:57:03.543 INFO:teuthology.orchestra.run.smithi002.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-08T15:57:03.543 INFO:teuthology.orchestra.run.smithi002.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-08T15:57:03.544 INFO:teuthology.orchestra.run.smithi002.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-08T15:57:03.575 INFO:teuthology.orchestra.run.smithi002.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-08T15:57:03.576 INFO:teuthology.orchestra.run.smithi002.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-08T15:57:03.586 INFO:teuthology.orchestra.run.smithi002.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-08T15:57:03.586 INFO:teuthology.orchestra.run.smithi002.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-08T15:57:03.586 INFO:teuthology.orchestra.run.smithi002.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-08T15:57:03.589 INFO:teuthology.orchestra.run.smithi002.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-08T15:57:03.590 INFO:teuthology.orchestra.run.smithi002.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-08T15:57:04.526 INFO:teuthology.orchestra.run.smithi002.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2517-g95a4ad3c-1jammy [3,216 kB] 2024-05-08T15:57:05.894 INFO:teuthology.orchestra.run.smithi002.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2517-g95a4ad3c-1jammy [711 kB] 2024-05-08T15:57:06.160 INFO:teuthology.orchestra.run.smithi002.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2517-g95a4ad3c-1jammy [356 kB] 2024-05-08T15:57:06.301 INFO:teuthology.orchestra.run.smithi002.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2517-g95a4ad3c-1jammy [32.8 kB] 2024-05-08T15:57:06.333 INFO:teuthology.orchestra.run.smithi002.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2517-g95a4ad3c-1jammy [180 kB] 2024-05-08T15:57:06.387 INFO:teuthology.orchestra.run.smithi002.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2517-g95a4ad3c-1jammy [67.1 kB] 2024-05-08T15:57:06.431 INFO:teuthology.orchestra.run.smithi002.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2517-g95a4ad3c-1jammy [330 kB] 2024-05-08T15:57:06.565 INFO:teuthology.orchestra.run.smithi002.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2517-g95a4ad3c-1jammy [6,779 kB] 2024-05-08T15:57:08.876 INFO:teuthology.orchestra.run.smithi002.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2517-g95a4ad3c-1jammy [112 kB] 2024-05-08T15:57:08.918 INFO:teuthology.orchestra.run.smithi002.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2517-g95a4ad3c-1jammy [466 kB] 2024-05-08T15:57:09.071 INFO:teuthology.orchestra.run.smithi002.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2517-g95a4ad3c-1jammy [25.6 MB] 2024-05-08T15:57:18.177 INFO:teuthology.orchestra.run.smithi002.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2517-g95a4ad3c-1jammy [5,085 kB] 2024-05-08T15:57:19.102 INFO:teuthology.orchestra.run.smithi002.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2517-g95a4ad3c-1jammy [244 kB] 2024-05-08T15:57:19.140 INFO:teuthology.orchestra.run.smithi002.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2517-g95a4ad3c-1jammy [124 kB] 2024-05-08T15:57:19.155 INFO:teuthology.orchestra.run.smithi002.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2517-g95a4ad3c-1jammy [1,526 kB] 2024-05-08T15:57:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2517-g95a4ad3c-1jammy [6,246 kB] 2024-05-08T15:57:20.546 INFO:teuthology.orchestra.run.smithi002.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2517-g95a4ad3c-1jammy [22.7 MB] 2024-05-08T15:57:24.798 INFO:teuthology.orchestra.run.smithi002.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2517-g95a4ad3c-1jammy [14.1 kB] 2024-05-08T15:57:24.798 INFO:teuthology.orchestra.run.smithi002.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2517-g95a4ad3c-1jammy [869 kB] 2024-05-08T15:57:25.012 INFO:teuthology.orchestra.run.smithi002.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2517-g95a4ad3c-1jammy [2,437 kB] 2024-05-08T15:57:25.657 INFO:teuthology.orchestra.run.smithi002.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2517-g95a4ad3c-1jammy [791 kB] 2024-05-08T15:57:25.913 INFO:teuthology.orchestra.run.smithi002.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2517-g95a4ad3c-1jammy [148 kB] 2024-05-08T15:57:25.916 INFO:teuthology.orchestra.run.smithi002.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2517-g95a4ad3c-1jammy [2,331 kB] 2024-05-08T15:57:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2517-g95a4ad3c-1jammy [8,625 kB] 2024-05-08T15:57:28.124 INFO:teuthology.orchestra.run.smithi002.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2517-g95a4ad3c-1jammy [14.2 kB] 2024-05-08T15:57:28.170 INFO:teuthology.orchestra.run.smithi002.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2517-g95a4ad3c-1jammy [50.7 MB] 2024-05-08T15:57:35.346 INFO:teuthology.orchestra.run.smithi002.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2517-g95a4ad3c-1jammy [121 kB] 2024-05-08T15:57:35.349 INFO:teuthology.orchestra.run.smithi002.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2517-g95a4ad3c-1jammy [38.9 kB] 2024-05-08T15:57:35.350 INFO:teuthology.orchestra.run.smithi002.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2517-g95a4ad3c-1jammy [13.4 MB] 2024-05-08T15:57:36.615 INFO:teuthology.orchestra.run.smithi002.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2517-g95a4ad3c-1jammy [91.6 kB] 2024-05-08T15:57:37.205 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 180 MB in 34s (5,226 kB/s) 2024-05-08T15:57:37.368 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-08T15:57:37.471 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-08T15:57:37.477 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-08T15:57:37.559 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-08T15:57:37.921 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-08T15:57:37.947 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-08T15:57:37.987 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-08T15:57:38.315 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-08T15:57:38.342 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-08T15:57:38.381 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-08T15:57:38.868 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-08T15:57:38.895 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:57:38.935 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:57:39.338 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-08T15:57:39.364 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:57:39.404 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:57:39.749 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-08T15:57:39.776 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:57:39.832 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:57:40.152 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-08T15:57:40.168 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-08T15:57:40.202 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-08T15:57:40.497 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librados2. 2024-05-08T15:57:40.523 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../007-librados2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:40.563 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librados2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:41.174 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librbd1. 2024-05-08T15:57:41.201 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../008-librbd1_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:41.241 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librbd1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:41.677 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs2. 2024-05-08T15:57:41.704 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:41.744 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:41.988 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rados. 2024-05-08T15:57:42.003 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:42.037 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rados (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:42.365 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-08T15:57:42.392 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:57:42.432 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-argparse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:42.684 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cephfs. 2024-05-08T15:57:42.700 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:42.743 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cephfs (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:43.054 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-08T15:57:43.080 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:57:43.120 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:43.415 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-08T15:57:43.431 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-08T15:57:43.465 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-08T15:57:43.753 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-prettytable. 2024-05-08T15:57:43.780 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-08T15:57:43.819 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-08T15:57:44.089 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rbd. 2024-05-08T15:57:44.105 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:44.147 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rbd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:44.433 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-08T15:57:44.460 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-08T15:57:44.500 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-08T15:57:44.777 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-08T15:57:44.804 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-08T15:57:44.852 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-08T15:57:45.147 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-08T15:57:45.173 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-08T15:57:45.213 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-08T15:57:45.516 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua5.1. 2024-05-08T15:57:45.543 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-08T15:57:45.584 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-08T15:57:45.894 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-any. 2024-05-08T15:57:45.921 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-08T15:57:45.961 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-08T15:57:46.197 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package zip. 2024-05-08T15:57:46.223 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-08T15:57:46.263 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking zip (3.0-12build2) ... 2024-05-08T15:57:46.733 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package unzip. 2024-05-08T15:57:46.760 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-08T15:57:46.801 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-08T15:57:47.162 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package luarocks. 2024-05-08T15:57:47.189 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-08T15:57:47.220 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-08T15:57:47.578 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librgw2. 2024-05-08T15:57:47.604 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../025-librgw2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:47.653 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librgw2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:48.197 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rgw. 2024-05-08T15:57:48.212 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:48.254 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:48.582 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-08T15:57:48.609 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-08T15:57:48.650 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-08T15:57:48.902 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libradosstriper1. 2024-05-08T15:57:48.928 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:48.960 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libradosstriper1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:49.380 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-common. 2024-05-08T15:57:49.407 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:49.447 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:50.868 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-base. 2024-05-08T15:57:50.898 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:57:50.961 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-base (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:57:51.514 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-08T15:57:51.541 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-08T15:57:51.581 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-08T15:57:51.983 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cheroot. 2024-05-08T15:57:51.999 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-08T15:57:52.033 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-08T15:57:52.362 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-08T15:57:52.389 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-08T15:57:52.428 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-08T15:57:52.731 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-08T15:57:52.758 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-08T15:57:52.798 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-08T15:57:53.067 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-08T15:57:53.094 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-08T15:57:53.125 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-08T15:57:53.412 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempora. 2024-05-08T15:57:53.439 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-08T15:57:53.478 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-08T15:57:53.789 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-portend. 2024-05-08T15:57:53.816 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-08T15:57:53.856 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-08T15:57:54.158 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-08T15:57:54.185 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-08T15:57:54.225 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-08T15:57:54.478 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-08T15:57:54.499 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-08T15:57:54.536 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-08T15:57:54.881 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-natsort. 2024-05-08T15:57:54.908 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-08T15:57:54.948 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-08T15:57:55.295 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-logutils. 2024-05-08T15:57:55.322 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-08T15:57:55.367 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-08T15:57:55.671 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-08T15:57:55.697 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-08T15:57:55.737 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-08T15:57:55.982 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-mako. 2024-05-08T15:57:56.008 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-08T15:57:56.048 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-08T15:57:56.384 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-08T15:57:56.412 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-08T15:57:56.451 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-08T15:57:56.754 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-08T15:57:56.781 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-08T15:57:56.820 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-08T15:57:57.215 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webob. 2024-05-08T15:57:57.242 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-08T15:57:57.273 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-08T15:57:57.534 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-waitress. 2024-05-08T15:57:57.562 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-08T15:57:57.596 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-08T15:57:57.887 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempita. 2024-05-08T15:57:57.914 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-08T15:57:57.954 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-08T15:57:58.240 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-paste. 2024-05-08T15:57:58.267 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-08T15:57:58.306 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-08T15:57:58.659 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-08T15:57:58.686 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-08T15:57:58.726 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-08T15:57:59.004 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-08T15:57:59.031 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-08T15:57:59.070 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-08T15:57:59.348 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webtest. 2024-05-08T15:57:59.375 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-08T15:57:59.415 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-08T15:57:59.709 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pecan. 2024-05-08T15:57:59.736 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-08T15:57:59.784 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-08T15:58:00.156 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-certifi. 2024-05-08T15:58:00.183 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-08T15:58:00.222 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-08T15:58:00.525 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-idna. 2024-05-08T15:58:00.552 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-08T15:58:00.591 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-08T15:58:00.844 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-urllib3. 2024-05-08T15:58:00.860 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-08T15:58:00.902 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-08T15:58:01.197 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests. 2024-05-08T15:58:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-08T15:58:01.264 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-08T15:58:01.541 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-08T15:58:01.565 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-08T15:58:01.600 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-08T15:58:01.894 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-08T15:58:01.910 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:01.952 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:02.350 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-08T15:58:02.376 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:02.416 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:02.685 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-08T15:58:02.712 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-08T15:58:02.743 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-08T15:58:03.013 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-openssl. 2024-05-08T15:58:03.040 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-08T15:58:03.079 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-08T15:58:03.416 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr. 2024-05-08T15:58:03.443 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:03.482 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:03.885 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mon. 2024-05-08T15:58:03.912 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:03.973 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mon (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:04.505 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-osd. 2024-05-08T15:58:04.532 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:04.572 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-osd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:05.619 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph. 2024-05-08T15:58:05.644 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../066-ceph_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:05.683 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:06.002 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-fuse. 2024-05-08T15:58:06.030 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:06.069 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:06.381 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mds. 2024-05-08T15:58:06.408 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:06.438 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mds (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:06.825 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package cephadm. 2024-05-08T15:58:06.842 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../069-cephadm_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:06.883 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:07.194 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-08T15:58:07.221 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-08T15:58:07.260 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-08T15:58:07.730 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-08T15:58:07.757 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-08T15:58:07.796 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-08T15:58:08.776 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-babel. 2024-05-08T15:58:08.803 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-08T15:58:08.834 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-08T15:58:09.121 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jinja2. 2024-05-08T15:58:09.148 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-08T15:58:09.179 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-08T15:58:09.474 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-08T15:58:09.501 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:09.540 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:09.801 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-08T15:58:09.817 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-08T15:58:09.851 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-08T15:58:10.113 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-routes. 2024-05-08T15:58:10.140 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-08T15:58:10.179 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-08T15:58:10.465 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-08T15:58:10.492 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:10.524 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:11.704 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-08T15:58:11.731 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-08T15:58:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-08T15:58:12.282 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-joblib. 2024-05-08T15:58:12.309 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-08T15:58:12.348 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-08T15:58:12.743 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-08T15:58:12.771 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-08T15:58:12.810 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-08T15:58:13.230 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn. 2024-05-08T15:58:13.257 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-08T15:58:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-08T15:58:13.962 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-08T15:58:13.990 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:14.114 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cachetools. 2024-05-08T15:58:15.054 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-08T15:58:15.093 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-08T15:58:15.337 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-08T15:58:15.365 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-08T15:58:15.395 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-08T15:58:15.682 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-08T15:58:15.710 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-08T15:58:15.749 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-08T15:58:16.060 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rsa. 2024-05-08T15:58:16.088 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-08T15:58:16.126 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-08T15:58:16.447 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-google-auth. 2024-05-08T15:58:16.474 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-08T15:58:16.513 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-08T15:58:16.799 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-08T15:58:16.816 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-08T15:58:16.849 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-08T15:58:17.119 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-websocket. 2024-05-08T15:58:17.146 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-08T15:58:17.185 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-08T15:58:17.506 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-08T15:58:17.533 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-08T15:58:17.572 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-08T15:58:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-08T15:58:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:18.663 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-08T15:58:18.691 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-08T15:58:18.730 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-08T15:58:19.033 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-08T15:58:19.061 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-08T15:58:19.099 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-08T15:58:19.335 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package jq. 2024-05-08T15:58:19.363 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-08T15:58:19.402 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-08T15:58:19.654 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package socat. 2024-05-08T15:58:19.670 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-08T15:58:19.712 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-08T15:58:20.001 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package xmlstarlet. 2024-05-08T15:58:20.017 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-08T15:58:20.059 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-08T15:58:20.413 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-test. 2024-05-08T15:58:20.441 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:20.479 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-test (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:22.976 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-volume. 2024-05-08T15:58:23.004 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:58:23.042 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-volume (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:23.562 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-08T15:58:23.590 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:23.630 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs-dev (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:23.957 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-08T15:58:23.985 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-08T15:58:24.023 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-08T15:58:24.353 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-08T15:58:24.381 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-08T15:58:24.419 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-08T15:58:24.715 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package nvme-cli. 2024-05-08T15:58:24.743 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-08T15:58:24.781 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-08T15:58:25.203 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package pkg-config. 2024-05-08T15:58:25.231 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-08T15:58:25.269 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-08T15:58:25.505 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-08T15:58:25.522 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-08T15:58:25.555 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-08T15:58:25.959 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-08T15:58:25.987 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-08T15:58:26.025 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-08T15:58:26.294 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastescript. 2024-05-08T15:58:26.322 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-08T15:58:26.361 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-08T15:58:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pluggy. 2024-05-08T15:58:26.709 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-08T15:58:26.747 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-08T15:58:27.042 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-psutil. 2024-05-08T15:58:27.070 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-08T15:58:27.109 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-08T15:58:27.428 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-py. 2024-05-08T15:58:27.456 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-08T15:58:27.519 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-08T15:58:27.839 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pygments. 2024-05-08T15:58:27.867 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-08T15:58:27.906 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-08T15:58:28.377 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-08T15:58:28.405 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-08T15:58:28.443 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-08T15:58:28.729 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-toml. 2024-05-08T15:58:28.757 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-08T15:58:28.795 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-08T15:58:29.031 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pytest. 2024-05-08T15:58:29.048 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-08T15:58:29.098 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-08T15:58:29.435 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplejson. 2024-05-08T15:58:29.463 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-08T15:58:29.501 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-08T15:58:29.779 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-08T15:58:29.796 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-08T15:58:29.837 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-08T15:58:30.541 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package radosgw. 2024-05-08T15:58:30.570 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../116-radosgw_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:30.608 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking radosgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:31.436 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package rbd-fuse. 2024-05-08T15:58:31.452 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:58:31.494 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking rbd-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:31.799 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-08T15:58:32.166 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-08T15:58:32.292 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-08T15:58:32.556 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-08T15:58:32.892 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-08T15:58:33.283 INFO:teuthology.orchestra.run.smithi002.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-08T15:58:33.360 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:34.068 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user cephadm....done 2024-05-08T15:58:34.216 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-08T15:58:34.500 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-08T15:58:34.768 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-08T15:58:34.894 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-08T15:58:35.154 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-08T15:58:35.396 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-08T15:58:35.506 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-08T15:58:35.808 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-08T15:58:36.168 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-08T15:58:36.427 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-08T15:58:36.712 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-08T15:58:36.938 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-08T15:58:37.172 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-08T15:58:37.440 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-08T15:58:37.691 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-argparse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:37.951 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-08T15:58:38.076 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-08T15:58:38.202 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-08T15:58:38.328 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-08T15:58:38.571 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-08T15:58:38.938 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-08T15:58:38.978 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-08T15:58:39.050 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-08T15:58:39.176 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-08T15:58:39.543 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-08T15:58:39.669 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-08T15:58:39.937 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-08T15:58:40.196 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-08T15:58:40.489 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up zip (3.0-12build2) ... 2024-05-08T15:58:40.615 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-08T15:58:41.309 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-08T15:58:41.577 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-08T15:58:41.719 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-08T15:58:41.845 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-08T15:58:42.138 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-08T15:58:42.514 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-08T15:58:42.774 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-08T15:58:43.176 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-08T15:58:43.468 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-08T15:58:43.828 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-08T15:58:44.079 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-08T15:58:44.180 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:44.465 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-idna (3.3-1) ... 2024-05-08T15:58:44.766 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-08T15:58:46.005 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-08T15:58:46.248 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:58:46.391 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-08T15:58:46.658 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-08T15:58:46.784 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-08T15:58:46.910 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-08T15:58:47.212 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-08T15:58:47.446 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-08T15:58:47.706 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:58:47.840 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-08T15:58:48.158 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-08T15:58:48.418 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-08T15:58:48.685 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-08T15:58:48.953 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-08T15:58:49.212 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-08T15:58:49.472 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-08T15:58:49.589 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-08T15:58:49.874 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-08T15:58:50.000 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-08T15:58:50.318 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-08T15:58:50.677 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-08T15:58:50.937 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-08T15:58:51.038 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-08T15:58:51.139 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-08T15:58:51.229 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-08T15:58:51.474 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-08T15:58:51.600 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-08T15:58:52.035 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-08T15:58:52.302 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-08T15:58:52.428 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-08T15:58:52.679 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:58:52.780 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-08T15:58:53.098 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-08T15:58:53.391 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-08T15:58:53.667 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-08T15:58:53.793 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-08T15:58:54.069 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-08T15:58:54.328 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-08T15:58:54.665 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-08T15:58:55.050 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-08T15:58:55.352 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-08T15:58:55.712 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-08T15:58:55.813 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librados2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:56.032 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-08T15:58:56.393 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:56.519 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libradosstriper1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:56.645 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-08T15:58:56.930 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librbd1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:57.047 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-modules-core (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:57.174 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-08T15:58:57.392 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-08T15:58:57.875 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs-dev (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:58.001 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rados (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:58.127 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librgw2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:58.253 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-08T15:58:59.549 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rbd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:59.675 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up rbd-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:59.801 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:58:59.927 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cephfs (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:00.053 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:00.563 INFO:teuthology.orchestra.run.smithi002.stdout:Adding group ceph....done 2024-05-08T15:59:00.847 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user ceph....done 2024-05-08T15:59:00.991 INFO:teuthology.orchestra.run.smithi002.stdout:Setting system user ceph properties....done 2024-05-08T15:59:01.006 INFO:teuthology.orchestra.run.smithi002.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-08T15:59:01.146 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-08T15:59:01.446 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-08T15:59:01.912 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-test (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:02.013 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up radosgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:02.487 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-08T15:59:02.488 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-08T15:59:02.965 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-base (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:03.315 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-08T15:59:03.794 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mds (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:04.040 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-08T15:59:04.040 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-08T15:59:04.502 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:04.711 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-08T15:59:04.712 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-08T15:59:05.228 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-osd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:05.526 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-08T15:59:05.526 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-08T15:59:06.038 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:06.164 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:06.324 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mon (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:06.550 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-08T15:59:06.550 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-08T15:59:07.033 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:07.176 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:07.277 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-dashboard (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:07.411 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-volume (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:59:07.788 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-08T15:59:07.889 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-08T15:59:08.042 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-08T15:59:09.167 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-08T15:59:09.811 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-08T15:59:09.816 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:09.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:59:10.040 DEBUG:teuthology.orchestra.run.smithi002:> dpkg-query -W -f '${Version}' ceph 2024-05-08T15:59:10.059 INFO:teuthology.orchestra.run.smithi002.stdout:19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:59:10.060 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:59:10.060 INFO:teuthology.task.install:The correct ceph version 19.0.0-2517-g95a4ad3c-1jammy is installed. 2024-05-08T15:59:10.061 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-08T15:59:10.062 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:10.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T15:59:10.117 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-08T15:59:10.118 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:10.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T15:59:10.173 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T15:59:10.228 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-08T15:59:10.228 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:10.228 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T15:59:10.284 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T15:59:10.340 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-08T15:59:10.340 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:10.340 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T15:59:10.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T15:59:10.452 INFO:teuthology.run_tasks:Running task ceph... 2024-05-08T15:59:10.531 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-08T15:59:10.531 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2024-05-08T15:59:10.545 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-08T15:59:10.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-08T15:59:10.599 INFO:tasks.ceph:Creating extra log directories... 2024-05-08T15:59:10.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-08T15:59:10.664 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-08T15:59:10.665 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-08T15:59:10.665 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631', 'branch': 'wip-lflores-testing-2-2024-05-07-1606-squid', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}', 'email': None, 'first_in_suite': False, 'job_id': '7697631', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lflores-testing-2-2024-05-07-1606-squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'}}, 'workunit': {'branch': 'wip-lflores-testing-2-2024-05-07-1606-squid', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'}}, 'owner': 'scheduled_lflores@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'seed': 8082, 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-lflores-testing-2-2024-05-07-1606-squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjJIzyZqhpoq7p/b5uN9kUUI/hwkd8ztS5LtUwbkEruEs+8X7SfR84X0yyKHIBtz2V1O1ILR45DfCoMfmcPudo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-08_14:59:36', 'tube': 'smithi', 'user': 'lflores', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-08T15:59:10.665 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-08T15:59:10.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2024-05-08T15:59:10.760 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:10.760 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-05-08T15:59:10.808 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-08T15:59:10.808 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-05-08 15:53:32.680709255 +0000 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-05-08 15:53:32.432711470 +0000 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-05-08 15:53:32.432711470 +0000 2024-05-08T15:59:10.857 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-05-08T15:59:10.857 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-08T15:59:10.910 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-05-08T15:59:10.911 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-05-08T15:59:10.911 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000287366 s, 1.8 MB/s 2024-05-08T15:59:10.912 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-08T15:59:10.962 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-05-08 15:53:33.156705004 +0000 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-05-08 15:53:32.908707219 +0000 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-05-08 15:53:32.908707219 +0000 2024-05-08T15:59:11.009 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-05-08T15:59:11.009 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-08T15:59:11.063 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-05-08T15:59:11.063 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-05-08T15:59:11.063 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000247193 s, 2.1 MB/s 2024-05-08T15:59:11.064 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-08T15:59:11.114 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-05-08T15:59:11.160 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-05-08 15:53:33.656700538 +0000 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-05-08 15:53:33.384702967 +0000 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-05-08 15:53:33.384702967 +0000 2024-05-08T15:59:11.161 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-05-08T15:59:11.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-08T15:59:11.215 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-05-08T15:59:11.215 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-05-08T15:59:11.215 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00031607 s, 1.6 MB/s 2024-05-08T15:59:11.216 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-08T15:59:11.267 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-05-08T15:59:11.316 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-08T15:59:11.316 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-05-08 15:53:34.124696358 +0000 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-05-08 15:53:33.868698645 +0000 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-05-08 15:53:33.868698645 +0000 2024-05-08T15:59:11.317 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2024-05-08T15:59:11.317 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-08T15:59:11.371 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-05-08T15:59:11.372 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-05-08T15:59:11.372 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000291677 s, 1.8 MB/s 2024-05-08T15:59:11.373 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-08T15:59:11.422 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:11.423 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-05-08T15:59:11.423 INFO:tasks.ceph:Generating config... 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd max object name len = 400 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-08T15:59:11.424 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-08T15:59:11.425 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-08T15:59:11.425 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-08T15:59:11.425 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-08T15:59:11.425 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-08T15:59:11.425 INFO:tasks.ceph:Setting up mon.a... 2024-05-08T15:59:11.425 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-08T15:59:11.496 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2024-05-08T15:59:11.506 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-08T15:59:11.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-08T15:59:11.640 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2')] 2024-05-08T15:59:11.640 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2', 'mon election default strategy': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-05-08T15:59:11.641 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:11.641 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T15:59:11.688 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:11.769 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:11.769 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid 84e991d3-12d8-4331-8afe-a495890a074b 2024-05-08T15:59:11.769 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = quincy 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 84e991d3-12d8-4331-8afe-a495890a074b 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2024-05-08T15:59:11.775039+0000 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-05-08T15:59:11.775039+0000 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 17 (quincy) 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 3 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-05-08T15:59:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-05-08T15:59:11.771 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-08T15:59:11.816 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 84e991d3-12d8-4331-8afe-a495890a074b... 2024-05-08T15:59:11.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-08T15:59:11.890 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-05-08T15:59:11.890 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.891 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-08T15:59:11.892 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.2 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: mon election default strategy = 3 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 84e991d3-12d8-4331-8afe-a495890a074b 2024-05-08T15:59:11.893 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2024-05-08T15:59:11.894 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2024-05-08T15:59:11.895 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd max object name len = 400 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd max object namespace len = 64 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore zero block detection = True 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify cached snaps = True 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify missing on start = True 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock override recovery settings = True 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue = debug_random 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: osd op queue cut off = debug_random 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-05-08T15:59:11.896 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2024-05-08T15:59:11.897 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2024-05-08T15:59:11.898 INFO:teuthology.orchestra.run.smithi002.stdout: mon scrub interval = 300 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-08T15:59:11.899 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2024-05-08T15:59:11.900 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-08T15:59:11.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-08T15:59:11.993 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-08T15:59:11.993 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:11.993 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-08T15:59:11.999 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:11.999 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-08T15:59:12.044 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:59:12.044 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:12.044 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-08T15:59:12.045 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-08T15:59:12.109 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:12.110 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:12.156 INFO:tasks.ceph:Setting up mon nodes... 2024-05-08T15:59:12.157 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-08T15:59:12.157 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-08T15:59:12.243 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-08T15:59:12.246 INFO:tasks.ceph:Setting up mds nodes... 2024-05-08T15:59:12.246 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-08T15:59:12.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-08T15:59:12.322 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-08T15:59:12.333 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-08T15:59:12.333 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-05-08T15:59:12.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-08T15:59:12.387 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:12.387 INFO:tasks.ceph:role: osd.0 2024-05-08T15:59:12.388 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:59:12.388 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T15:59:12.444 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T15:59:12.468 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-05-08T15:59:12.470 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-05-08T15:59:12.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-08T15:59:12.529 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-08T15:59:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:59:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-05-08T15:59:12.581 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-08T15:59:12.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:12.634 INFO:tasks.ceph:role: osd.1 2024-05-08T15:59:12.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:59:12.634 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-08T15:59:12.689 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T15:59:12.689 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T15:59:12.689 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T15:59:12.689 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T15:59:12.690 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T15:59:12.714 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-05-08T15:59:12.715 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-05-08T15:59:12.716 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-08T15:59:12.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-08T15:59:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:59:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-05-08T15:59:12.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-08T15:59:12.884 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:12.884 INFO:tasks.ceph:role: osd.2 2024-05-08T15:59:12.884 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:59:12.884 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T15:59:12.942 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T15:59:12.943 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T15:59:12.965 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-05-08T15:59:12.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-05-08T15:59:12.967 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-08T15:59:13.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-08T15:59:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:59:13.081 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-05-08T15:59:13.081 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-08T15:59:13.134 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.134 INFO:tasks.ceph:role: osd.3 2024-05-08T15:59:13.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T15:59:13.134 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-08T15:59:13.188 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-08T15:59:13.188 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-08T15:59:13.189 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-08T15:59:13.213 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-05-08T15:59:13.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-05-08T15:59:13.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-08T15:59:13.272 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-08T15:59:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:59:13.325 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2024-05-08T15:59:13.325 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-08T15:59:13.378 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.378 INFO:tasks.ceph:role: osd.4 2024-05-08T15:59:13.378 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-08T15:59:13.431 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.432 INFO:tasks.ceph:role: osd.5 2024-05-08T15:59:13.432 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-08T15:59:13.488 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.488 INFO:tasks.ceph:role: osd.6 2024-05-08T15:59:13.488 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-08T15:59:13.544 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.544 INFO:tasks.ceph:role: osd.7 2024-05-08T15:59:13.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-05-08T15:59:13.601 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.601 INFO:tasks.ceph:role: osd.8 2024-05-08T15:59:13.601 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-05-08T15:59:13.660 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-08T15:59:13.661 INFO:tasks.ceph:role: osd.9 2024-05-08T15:59:13.661 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-08T15:59:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-08T15:59:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T15:59:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T15:59:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-08T15:59:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:13.777+0000 7f1a1dde87c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-08T15:59:15.943 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-08T15:59:15.957 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:16.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.041+0000 7f95509d77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-08T15:59:16.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.041+0000 7f95509d77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-08T15:59:16.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.041+0000 7f95509d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T15:59:16.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.041+0000 7f95509d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T15:59:16.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.041+0000 7f95509d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-08T15:59:16.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:16.045+0000 7f95509d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-08T15:59:18.219 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-08T15:59:18.277 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-08T15:59:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-08T15:59:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T15:59:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T15:59:18.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-08T15:59:18.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:18.369+0000 7fe4354687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-08T15:59:20.542 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-08T15:59:20.597 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:20.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-08T15:59:20.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-08T15:59:20.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T15:59:20.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T15:59:20.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-08T15:59:20.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:20.684+0000 7f54ca9ad7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-08T15:59:22.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-08T15:59:22.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-08T15:59:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-08T15:59:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T15:59:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T15:59:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-08T15:59:23.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:23.008+0000 7fbbb42c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-08T15:59:25.175 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-08T15:59:25.237 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-08T15:59:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:25.320+0000 7fad22a537c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-08T15:59:27.503 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-08T15:59:27.557 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-08T15:59:27.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:27.640+0000 7f44ea5bd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-08T15:59:29.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-08T15:59:29.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.952+0000 7fed4dadf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.952+0000 7fed4dadf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.952+0000 7fed4dadf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.952+0000 7fed4dadf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.952+0000 7fed4dadf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-08T15:59:29.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:29.956+0000 7fed4dadf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-08T15:59:32.127 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-08T15:59:32.189 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-05-08T15:59:32.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:32.272+0000 7fb7a6bf67c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-05-08T15:59:34.455 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-05-08T15:59:34.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-05-08T15:59:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-05-08T15:59:34.596+0000 7fa32a0947c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-05-08T15:59:36.791 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-05-08T15:59:36.848 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-08T15:59:36.848 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:36.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-08T15:59:36.903 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:36.903 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-08T15:59:36.959 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:36.959 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-08T15:59:37.016 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-08T15:59:37.072 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.072 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-08T15:59:37.128 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-08T15:59:37.188 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.188 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-08T15:59:37.244 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-08T15:59:37.304 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-08T15:59:37.364 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-05-08T15:59:37.420 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.420 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-05-08T15:59:37.475 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:37.475 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-08T15:59:37.524 INFO:tasks.ceph:Adding keys to all mons... 2024-05-08T15:59:37.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBQoTtm3vrZDhAAE/bDYSWqjr2/1vF7+FBwdw== 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBRoTtm5lJsLhAAGifaRpugN/wlUWcxBZk1LA== 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBUoTtmAYSvAhAAmSUPIjfAAysqmHqG2mrT2A== 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBWoTtmGBQJFhAAP6g3C4e8cIicWFmfhJS4PQ== 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.3] 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBYoTtmhyDkKBAAmjepnmy6HIP4/7T6Tpp4Dw== 2024-05-08T15:59:37.579 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.4] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBboTtmS+TDABAANmtxCVX4zOakkmKfdqXezw== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.5] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBdoTtmNHhZExAA4lmWBZRyEC9moyLJUgapOQ== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.6] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBfoTtm0bFnJhAAugpjNAtpiZsP7o4j1XM3eg== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.7] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBhoTtmf0QNORAAJTOPlg4j41khUnUqm1DOsQ== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.8] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBkoTtm6VN/EBAAM4dyTel9MAKmjKrPNEPvdg== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.9] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBmoTtmlfnQIxAApcOS75eQV38ReF0UNy9SnQ== 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-05-08T15:59:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBQoTtmrWCMExAAdnmpHTpNu6SgQ4PLYHgRTQ== 2024-05-08T15:59:37.581 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-08T15:59:37.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:37.713 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:37.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:37.884 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:37.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.061 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.402 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-08T15:59:38.489 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-08T15:59:38.582 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-08T15:59:38.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-08T15:59:38.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-08T15:59:38.681 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-08T15:59:38.736 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-08T15:59:38.786 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-08T15:59:38.787 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-08T15:59:38.787 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-08T15:59:38.829 INFO:tasks.ceph.mon.a:Started 2024-05-08T15:59:38.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph config set mgr mgr_pool false --force 2024-05-08T15:59:39.114 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-08T15:59:39.114 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-08T15:59:39.114 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-08T15:59:39.116 INFO:tasks.ceph.mgr.x:Started 2024-05-08T15:59:39.116 INFO:tasks.ceph:Setting crush tunables to default 2024-05-08T15:59:39.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2024-05-08T15:59:39.314 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-05-08T15:59:39.332 INFO:tasks.ceph:check_enable_crimson: False 2024-05-08T15:59:39.333 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-08T15:59:39.333 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-08T15:59:39.344 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.344 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-08T15:59:39.402 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-08T15:59:39.410 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:39.412+0000 7f1c11b31180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-08T15:59:39.421 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.421 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-08T15:59:39.475 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-08T15:59:39.532 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.532 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-08T15:59:39.588 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.589 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-08T15:59:39.644 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.644 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-08T15:59:39.700 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.700 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-05-08T15:59:39.757 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T15:59:39.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-05-08T15:59:39.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3 0 2024-05-08T15:59:39.967 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:39.968+0000 7f1c11b31180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-08T15:59:40.068 INFO:teuthology.orchestra.run.smithi002.stdout:0 2024-05-08T15:59:40.084 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new bccff570-697e-41bb-a25e-485c4f834c2e 1 2024-05-08T15:59:40.157 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:40.160+0000 7f1c11b31180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-08T15:59:40.285 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-05-08T15:59:40.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 0c855c70-a738-4ecd-acd2-7df525160975 2 2024-05-08T15:59:40.492 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-05-08T15:59:40.508 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 54600c26-8a38-47dc-932a-d17e98b047cc 3 2024-05-08T15:59:40.695 INFO:teuthology.orchestra.run.smithi002.stdout:3 2024-05-08T15:59:40.712 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new ddca8acb-7e0e-4768-9b4c-3b1fecb9c240 4 2024-05-08T15:59:40.737 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:40.740+0000 7f1c11b31180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-08T15:59:40.796 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:40.800+0000 7f1c11b31180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-08T15:59:40.864 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:40.868+0000 7f1c11b31180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-08T15:59:40.983 INFO:teuthology.orchestra.run.smithi002.stdout:4 2024-05-08T15:59:40.993 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:40.996+0000 7f1c11b31180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-08T15:59:41.000 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 7a051df4-19a4-4c95-8569-74438658105d 5 2024-05-08T15:59:41.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:41.052+0000 7f1c11b31180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-08T15:59:41.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:41.132+0000 7f1c11b31180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-08T15:59:41.187 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:41.192+0000 7f1c11b31180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-08T15:59:41.241 INFO:teuthology.orchestra.run.smithi002.stdout:5 2024-05-08T15:59:41.256 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab 6 2024-05-08T15:59:41.346 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:41.348+0000 7f1c11b31180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-08T15:59:41.412 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:41.416+0000 7f1c11b31180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-08T15:59:41.453 INFO:teuthology.orchestra.run.smithi002.stdout:6 2024-05-08T15:59:41.469 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 448ff819-e3da-47db-8423-873815a23025 7 2024-05-08T15:59:41.660 INFO:teuthology.orchestra.run.smithi002.stdout:7 2024-05-08T15:59:41.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 908caf6f-0199-481f-8fef-810c6bbec8ba 8 2024-05-08T15:59:41.897 INFO:teuthology.orchestra.run.smithi002.stdout:8 2024-05-08T15:59:41.912 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 0379f6b0-ec63-487d-92c6-866b04ac385d 9 2024-05-08T15:59:41.986 INFO:tasks.ceph.mgr.x.smithi002.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-08T15:59:41.986 INFO:tasks.ceph.mgr.x.smithi002.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-08T15:59:41.986 INFO:tasks.ceph.mgr.x.smithi002.stderr: from numpy import show_config as show_numpy_config 2024-05-08T15:59:42.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:42.032+0000 7f1c11b31180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-08T15:59:42.110 INFO:teuthology.orchestra.run.smithi002.stdout:9 2024-05-08T15:59:42.126 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-08T15:59:42.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-08T15:59:42.128 INFO:tasks.ceph.osd.0:Started 2024-05-08T15:59:42.128 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-08T15:59:42.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-08T15:59:42.130 INFO:tasks.ceph.osd.1:Started 2024-05-08T15:59:42.130 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-08T15:59:42.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-08T15:59:42.132 INFO:tasks.ceph.osd.2:Started 2024-05-08T15:59:42.132 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-08T15:59:42.132 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-08T15:59:42.134 INFO:tasks.ceph.osd.3:Started 2024-05-08T15:59:42.134 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-08T15:59:42.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-08T15:59:42.136 INFO:tasks.ceph.osd.4:Started 2024-05-08T15:59:42.136 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-08T15:59:42.136 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-08T15:59:42.138 INFO:tasks.ceph.osd.5:Started 2024-05-08T15:59:42.138 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-08T15:59:42.139 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-08T15:59:42.141 INFO:tasks.ceph.osd.6:Started 2024-05-08T15:59:42.141 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-08T15:59:42.141 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-08T15:59:42.143 INFO:tasks.ceph.osd.7:Started 2024-05-08T15:59:42.143 INFO:tasks.ceph.osd.8:Restarting daemon 2024-05-08T15:59:42.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-05-08T15:59:42.145 INFO:tasks.ceph.osd.8:Started 2024-05-08T15:59:42.145 INFO:tasks.ceph.osd.9:Restarting daemon 2024-05-08T15:59:42.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-05-08T15:59:42.147 INFO:tasks.ceph.osd.9:Started 2024-05-08T15:59:42.148 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T15:59:42.208 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:42.212+0000 7f1c11b31180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-08T15:59:42.412 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:42.413 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T15:59:42.114535+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T15:59:42.417 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:42.420+0000 7f1c11b31180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-08T15:59:42.428 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-08T15:59:42.428 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-08T15:59:42.485 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:42.488+0000 7f1c11b31180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-08T15:59:42.564 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-05-08T15:59:42.568+0000 7fb2ea9c57c0 -1 Falling back to public interface 2024-05-08T15:59:42.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T15:59:42.572+0000 7f3e64e9a7c0 -1 Falling back to public interface 2024-05-08T15:59:42.580 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T15:59:42.584+0000 7f41fbc587c0 -1 Falling back to public interface 2024-05-08T15:59:42.580 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T15:59:42.584+0000 7f4f4316e7c0 -1 Falling back to public interface 2024-05-08T15:59:42.632 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:42.636+0000 7f1c11b31180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-08T15:59:42.797 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T15:59:42.800+0000 7f7738f387c0 -1 Falling back to public interface 2024-05-08T15:59:42.812 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T15:59:42.816+0000 7fa3882e57c0 -1 Falling back to public interface 2024-05-08T15:59:42.820 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-05-08T15:59:42.824+0000 7f5c63c247c0 -1 Falling back to public interface 2024-05-08T15:59:42.820 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T15:59:42.824+0000 7fe1289a07c0 -1 Falling back to public interface 2024-05-08T15:59:42.821 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T15:59:42.824+0000 7f9855dd27c0 -1 Falling back to public interface 2024-05-08T15:59:42.821 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T15:59:42.824+0000 7fc832ddd7c0 -1 Falling back to public interface 2024-05-08T15:59:43.286 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:43.288+0000 7f1c11b31180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-08T15:59:43.364 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:43.368+0000 7f1c11b31180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-08T15:59:43.435 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-08T15:59:43.558 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:43.560+0000 7f1c11b31180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-08T15:59:43.606 INFO:teuthology.misc.health.smithi002.stdout: 2024-05-08T15:59:43.606 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":12,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T15:59:42.114535+0000","last_up_change":"0.000000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T15:59:43.621 DEBUG:teuthology.misc:0 of 10 OSDs are up 2024-05-08T15:59:43.637 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:43.640+0000 7f1c11b31180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-08T15:59:43.940 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:43.944+0000 7f1c11b31180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-08T15:59:44.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:44.008+0000 7f1c11b31180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-08T15:59:45.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2024-05-08T15:59:45.040+0000 7f1bb8f79640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-08T15:59:45.473 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T15:59:45.476+0000 7fa3882e57c0 -1 osd.0 0 log_to_monitors true 2024-05-08T15:59:45.611 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T15:59:45.616+0000 7f7738f387c0 -1 osd.6 0 log_to_monitors true 2024-05-08T15:59:45.642 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-05-08T15:59:45.644+0000 7f5c63c247c0 -1 osd.5 0 log_to_monitors true 2024-05-08T15:59:45.644 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T15:59:45.648+0000 7f3e64e9a7c0 -1 osd.1 0 log_to_monitors true 2024-05-08T15:59:45.666 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T15:59:45.668+0000 7f4f4316e7c0 -1 osd.3 0 log_to_monitors true 2024-05-08T15:59:45.705 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T15:59:45.708+0000 7f41fbc587c0 -1 osd.4 0 log_to_monitors true 2024-05-08T15:59:45.716 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-05-08T15:59:45.720+0000 7fb2ea9c57c0 -1 osd.2 0 log_to_monitors true 2024-05-08T15:59:45.878 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T15:59:45.880+0000 7fe1289a07c0 -1 osd.9 0 log_to_monitors true 2024-05-08T15:59:45.957 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T15:59:45.960+0000 7fc832ddd7c0 -1 osd.8 0 log_to_monitors true 2024-05-08T15:59:45.957 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T15:59:45.960+0000 7f9855dd27c0 -1 osd.7 0 log_to_monitors true 2024-05-08T15:59:47.061 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T15:59:47.064+0000 7f41f7c01640 -1 osd.4 0 waiting for initial osdmap 2024-05-08T15:59:47.062 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T15:59:47.064+0000 7f3e60e43640 -1 osd.1 0 waiting for initial osdmap 2024-05-08T15:59:47.063 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T15:59:47.064+0000 7fc82ed86640 -1 osd.8 0 waiting for initial osdmap 2024-05-08T15:59:47.063 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-05-08T15:59:47.064+0000 7fb2e696e640 -1 osd.2 0 waiting for initial osdmap 2024-05-08T15:59:47.119 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-05-08T15:59:47.120+0000 7fb2e27ab640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:47.146 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T15:59:47.148+0000 7f3e5cc80640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:47.175 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T15:59:47.176+0000 7f41f3a3e640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:47.216 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T15:59:47.220+0000 7fc82abc3640 -1 osd.8 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:47.959 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T15:59:47.964+0000 7f4f3f117640 -1 osd.3 0 waiting for initial osdmap 2024-05-08T15:59:47.971 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T15:59:47.972+0000 7fa38428e640 -1 osd.0 0 waiting for initial osdmap 2024-05-08T15:59:47.986 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T15:59:47.988+0000 7f4f36f4c640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:48.009 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T15:59:48.012+0000 7fa37c0c3640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:48.606 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T15:59:48.608+0000 7f7734ee1640 -1 osd.6 0 waiting for initial osdmap 2024-05-08T15:59:48.644 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T15:59:48.648+0000 7f772cd16640 -1 osd.6 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:48.644 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T15:59:48.648+0000 7f9851d7b640 -1 osd.7 0 waiting for initial osdmap 2024-05-08T15:59:48.648 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T15:59:48.652+0000 7fe124949640 -1 osd.9 0 waiting for initial osdmap 2024-05-08T15:59:48.649 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-05-08T15:59:48.652+0000 7f5c5fbcd640 -1 osd.5 0 waiting for initial osdmap 2024-05-08T15:59:48.668 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T15:59:48.672+0000 7f9849bb0640 -1 osd.7 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:48.676 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-05-08T15:59:48.680+0000 7f5c57a02640 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:48.676 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T15:59:48.680+0000 7fe11c77e640 -1 osd.9 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T15:59:50.628 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-08T15:59:50.939 INFO:teuthology.misc.health.smithi002.stdout: 2024-05-08T15:59:50.940 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":17,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T15:59:50.135953+0000","last_up_change":"2024-05-08T15:59:49.129221+0000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6841","nonce":2047446632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6843","nonce":2047446632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6847","nonce":2047446632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6845","nonce":2047446632}]},"public_addr":"172.21.15.2:6841/2047446632","cluster_addr":"172.21.15.2:6843/2047446632","heartbeat_back_addr":"172.21.15.2:6847/2047446632","heartbeat_front_addr":"172.21.15.2:6845/2047446632","state":["exists","up"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509145648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6811","nonce":3509145648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6815","nonce":3509145648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6813","nonce":3509145648}]},"public_addr":"172.21.15.2:6809/3509145648","cluster_addr":"172.21.15.2:6811/3509145648","heartbeat_back_addr":"172.21.15.2:6815/3509145648","heartbeat_front_addr":"172.21.15.2:6813/3509145648","state":["exists","up"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6801","nonce":4107948721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6803","nonce":4107948721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6807","nonce":4107948721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6805","nonce":4107948721}]},"public_addr":"172.21.15.2:6801/4107948721","cluster_addr":"172.21.15.2:6803/4107948721","heartbeat_back_addr":"172.21.15.2:6807/4107948721","heartbeat_front_addr":"172.21.15.2:6805/4107948721","state":["exists","up"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6821","nonce":607357490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6827","nonce":607357490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6831","nonce":607357490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6829","nonce":607357490}]},"public_addr":"172.21.15.2:6821/607357490","cluster_addr":"172.21.15.2:6827/607357490","heartbeat_back_addr":"172.21.15.2:6831/607357490","heartbeat_front_addr":"172.21.15.2:6829/607357490","state":["exists","up"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6817","nonce":3384451429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6820","nonce":3384451429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6826","nonce":3384451429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6823","nonce":3384451429}]},"public_addr":"172.21.15.2:6817/3384451429","cluster_addr":"172.21.15.2:6820/3384451429","heartbeat_back_addr":"172.21.15.2:6826/3384451429","heartbeat_front_addr":"172.21.15.2:6823/3384451429","state":["exists","up"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6852","nonce":364873758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6856","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6861","nonce":364873758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6872","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6876","nonce":364873758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6864","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6868","nonce":364873758}]},"public_addr":"172.21.15.2:6852/364873758","cluster_addr":"172.21.15.2:6861/364873758","heartbeat_back_addr":"172.21.15.2:6876/364873758","heartbeat_front_addr":"172.21.15.2:6868/364873758","state":["exists","up"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6833","nonce":2323196745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6835","nonce":2323196745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6839","nonce":2323196745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6837","nonce":2323196745}]},"public_addr":"172.21.15.2:6833/2323196745","cluster_addr":"172.21.15.2:6835/2323196745","heartbeat_back_addr":"172.21.15.2:6839/2323196745","heartbeat_front_addr":"172.21.15.2:6837/2323196745","state":["exists","up"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6854","nonce":672606001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6857","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6860","nonce":672606001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6873","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6877","nonce":672606001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6865","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6869","nonce":672606001}]},"public_addr":"172.21.15.2:6854/672606001","cluster_addr":"172.21.15.2:6860/672606001","heartbeat_back_addr":"172.21.15.2:6877/672606001","heartbeat_front_addr":"172.21.15.2:6869/672606001","state":["exists","up"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6851","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6855","nonce":1901204544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6863","nonce":1901204544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6875","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6879","nonce":1901204544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6871","nonce":1901204544}]},"public_addr":"172.21.15.2:6855/1901204544","cluster_addr":"172.21.15.2:6863/1901204544","heartbeat_back_addr":"172.21.15.2:6879/1901204544","heartbeat_front_addr":"172.21.15.2:6871/1901204544","state":["exists","up"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6849","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6853","nonce":1516429068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6862","nonce":1516429068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6878","nonce":1516429068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6870","nonce":1516429068}]},"public_addr":"172.21.15.2:6853/1516429068","cluster_addr":"172.21.15.2:6862/1516429068","heartbeat_back_addr":"172.21.15.2:6878/1516429068","heartbeat_front_addr":"172.21.15.2:6870/1516429068","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.506163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.612784+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.677431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.720569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.731739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.625047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.663568+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.965907+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.988031+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.891542+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T15:59:50.957 DEBUG:teuthology.misc:10 of 10 OSDs are up 2024-05-08T15:59:50.957 INFO:tasks.ceph:Creating RBD pool 2024-05-08T15:59:50.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-08T15:59:52.141 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2024-05-08T15:59:52.165 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-08T15:59:53.142 INFO:teuthology.orchestra.run.smithi002.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-08T15:59:53.173 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-08T15:59:53.174 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-08T15:59:53.174 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-08T15:59:53.174 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-08T15:59:53.174 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-08T15:59:53.517 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:53.535 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6880","nonce":2198455709},{"type":"v1","addr":"172.21.15.2:6881","nonce":2198455709}]},"active_addr":"172.21.15.2:6881/2198455709","active_change":"2024-05-08T15:59:44.031521+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2214899844}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2849064687}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2894507934}]}]} 2024-05-08T15:59:53.537 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-08T15:59:53.537 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-08T15:59:53.537 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T15:59:53.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:53.833 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":19,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T15:59:53.146622+0000","last_up_change":"2024-05-08T15:59:49.129221+0000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-08T15:59:51.300852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6841","nonce":2047446632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6843","nonce":2047446632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6847","nonce":2047446632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6845","nonce":2047446632}]},"public_addr":"172.21.15.2:6841/2047446632","cluster_addr":"172.21.15.2:6843/2047446632","heartbeat_back_addr":"172.21.15.2:6847/2047446632","heartbeat_front_addr":"172.21.15.2:6845/2047446632","state":["exists","up"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509145648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6811","nonce":3509145648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6815","nonce":3509145648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6813","nonce":3509145648}]},"public_addr":"172.21.15.2:6809/3509145648","cluster_addr":"172.21.15.2:6811/3509145648","heartbeat_back_addr":"172.21.15.2:6815/3509145648","heartbeat_front_addr":"172.21.15.2:6813/3509145648","state":["exists","up"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6801","nonce":4107948721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6803","nonce":4107948721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6807","nonce":4107948721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6805","nonce":4107948721}]},"public_addr":"172.21.15.2:6801/4107948721","cluster_addr":"172.21.15.2:6803/4107948721","heartbeat_back_addr":"172.21.15.2:6807/4107948721","heartbeat_front_addr":"172.21.15.2:6805/4107948721","state":["exists","up"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6821","nonce":607357490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6827","nonce":607357490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6831","nonce":607357490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6829","nonce":607357490}]},"public_addr":"172.21.15.2:6821/607357490","cluster_addr":"172.21.15.2:6827/607357490","heartbeat_back_addr":"172.21.15.2:6831/607357490","heartbeat_front_addr":"172.21.15.2:6829/607357490","state":["exists","up"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6817","nonce":3384451429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6820","nonce":3384451429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6826","nonce":3384451429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6823","nonce":3384451429}]},"public_addr":"172.21.15.2:6817/3384451429","cluster_addr":"172.21.15.2:6820/3384451429","heartbeat_back_addr":"172.21.15.2:6826/3384451429","heartbeat_front_addr":"172.21.15.2:6823/3384451429","state":["exists","up"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6852","nonce":364873758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6856","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6861","nonce":364873758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6872","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6876","nonce":364873758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6864","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6868","nonce":364873758}]},"public_addr":"172.21.15.2:6852/364873758","cluster_addr":"172.21.15.2:6861/364873758","heartbeat_back_addr":"172.21.15.2:6876/364873758","heartbeat_front_addr":"172.21.15.2:6868/364873758","state":["exists","up"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6833","nonce":2323196745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6835","nonce":2323196745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6839","nonce":2323196745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6837","nonce":2323196745}]},"public_addr":"172.21.15.2:6833/2323196745","cluster_addr":"172.21.15.2:6835/2323196745","heartbeat_back_addr":"172.21.15.2:6839/2323196745","heartbeat_front_addr":"172.21.15.2:6837/2323196745","state":["exists","up"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6854","nonce":672606001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6857","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6860","nonce":672606001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6873","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6877","nonce":672606001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6865","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6869","nonce":672606001}]},"public_addr":"172.21.15.2:6854/672606001","cluster_addr":"172.21.15.2:6860/672606001","heartbeat_back_addr":"172.21.15.2:6877/672606001","heartbeat_front_addr":"172.21.15.2:6869/672606001","state":["exists","up"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6851","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6855","nonce":1901204544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6863","nonce":1901204544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6875","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6879","nonce":1901204544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6871","nonce":1901204544}]},"public_addr":"172.21.15.2:6855/1901204544","cluster_addr":"172.21.15.2:6863/1901204544","heartbeat_back_addr":"172.21.15.2:6879/1901204544","heartbeat_front_addr":"172.21.15.2:6871/1901204544","state":["exists","up"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6849","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6853","nonce":1516429068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6862","nonce":1516429068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6878","nonce":1516429068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6870","nonce":1516429068}]},"public_addr":"172.21.15.2:6853/1516429068","cluster_addr":"172.21.15.2:6862/1516429068","heartbeat_back_addr":"172.21.15.2:6878/1516429068","heartbeat_front_addr":"172.21.15.2:6870/1516429068","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.506163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.612784+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.677431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.720569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.731739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.625047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.663568+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.965907+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.988031+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.891542+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T15:59:53.851 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-08T15:59:53.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T15:59:54.142 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:54.143 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":19,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T15:59:53.146622+0000","last_up_change":"2024-05-08T15:59:49.129221+0000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-08T15:59:51.300852+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.89999997615814209,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6841","nonce":2047446632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6843","nonce":2047446632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6847","nonce":2047446632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6845","nonce":2047446632}]},"public_addr":"172.21.15.2:6841/2047446632","cluster_addr":"172.21.15.2:6843/2047446632","heartbeat_back_addr":"172.21.15.2:6847/2047446632","heartbeat_front_addr":"172.21.15.2:6845/2047446632","state":["exists","up"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509145648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6811","nonce":3509145648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6815","nonce":3509145648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6813","nonce":3509145648}]},"public_addr":"172.21.15.2:6809/3509145648","cluster_addr":"172.21.15.2:6811/3509145648","heartbeat_back_addr":"172.21.15.2:6815/3509145648","heartbeat_front_addr":"172.21.15.2:6813/3509145648","state":["exists","up"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6801","nonce":4107948721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6803","nonce":4107948721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6807","nonce":4107948721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6805","nonce":4107948721}]},"public_addr":"172.21.15.2:6801/4107948721","cluster_addr":"172.21.15.2:6803/4107948721","heartbeat_back_addr":"172.21.15.2:6807/4107948721","heartbeat_front_addr":"172.21.15.2:6805/4107948721","state":["exists","up"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6821","nonce":607357490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6827","nonce":607357490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6831","nonce":607357490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6829","nonce":607357490}]},"public_addr":"172.21.15.2:6821/607357490","cluster_addr":"172.21.15.2:6827/607357490","heartbeat_back_addr":"172.21.15.2:6831/607357490","heartbeat_front_addr":"172.21.15.2:6829/607357490","state":["exists","up"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6817","nonce":3384451429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6820","nonce":3384451429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6826","nonce":3384451429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6823","nonce":3384451429}]},"public_addr":"172.21.15.2:6817/3384451429","cluster_addr":"172.21.15.2:6820/3384451429","heartbeat_back_addr":"172.21.15.2:6826/3384451429","heartbeat_front_addr":"172.21.15.2:6823/3384451429","state":["exists","up"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6852","nonce":364873758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6856","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6861","nonce":364873758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6872","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6876","nonce":364873758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6864","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6868","nonce":364873758}]},"public_addr":"172.21.15.2:6852/364873758","cluster_addr":"172.21.15.2:6861/364873758","heartbeat_back_addr":"172.21.15.2:6876/364873758","heartbeat_front_addr":"172.21.15.2:6868/364873758","state":["exists","up"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6833","nonce":2323196745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6835","nonce":2323196745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6839","nonce":2323196745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6837","nonce":2323196745}]},"public_addr":"172.21.15.2:6833/2323196745","cluster_addr":"172.21.15.2:6835/2323196745","heartbeat_back_addr":"172.21.15.2:6839/2323196745","heartbeat_front_addr":"172.21.15.2:6837/2323196745","state":["exists","up"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6854","nonce":672606001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6857","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6860","nonce":672606001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6873","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6877","nonce":672606001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6865","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6869","nonce":672606001}]},"public_addr":"172.21.15.2:6854/672606001","cluster_addr":"172.21.15.2:6860/672606001","heartbeat_back_addr":"172.21.15.2:6877/672606001","heartbeat_front_addr":"172.21.15.2:6869/672606001","state":["exists","up"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6851","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6855","nonce":1901204544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6863","nonce":1901204544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6875","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6879","nonce":1901204544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6871","nonce":1901204544}]},"public_addr":"172.21.15.2:6855/1901204544","cluster_addr":"172.21.15.2:6863/1901204544","heartbeat_back_addr":"172.21.15.2:6879/1901204544","heartbeat_front_addr":"172.21.15.2:6871/1901204544","state":["exists","up"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6849","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6853","nonce":1516429068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6862","nonce":1516429068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6878","nonce":1516429068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6870","nonce":1516429068}]},"public_addr":"172.21.15.2:6853/1516429068","cluster_addr":"172.21.15.2:6862/1516429068","heartbeat_back_addr":"172.21.15.2:6878/1516429068","heartbeat_front_addr":"172.21.15.2:6870/1516429068","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.506163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.612784+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.677431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.720569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.731739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.625047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.663568+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.965907+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.988031+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.891542+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T15:59:54.160 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-08T15:59:54.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-08T15:59:54.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-08T15:59:54.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-08T15:59:54.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-08T15:59:54.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-08T15:59:54.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-08T15:59:54.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-08T15:59:54.164 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-05-08T15:59:54.164 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-05-08T15:59:54.445 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:54.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-08T15:59:54.450 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-05-08T15:59:54.451 INFO:teuthology.orchestra.run.smithi002.stdout:68719476739 2024-05-08T15:59:54.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-08T15:59:54.452 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-08T15:59:54.460 INFO:teuthology.orchestra.run.smithi002.stdout:68719476740 2024-05-08T15:59:54.461 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-08T15:59:54.495 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:54.496 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-08T15:59:54.526 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:54.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-08T15:59:54.527 INFO:teuthology.orchestra.run.smithi002.stdout:68719476740 2024-05-08T15:59:54.527 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-05-08T15:59:54.536 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:54.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-05-08T15:59:54.539 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:54.540 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-08T15:59:54.558 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:54.559 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-08T15:59:55.024 INFO:teuthology.orchestra.run.smithi002.stdout:68719476740 2024-05-08T15:59:55.063 INFO:teuthology.orchestra.run.smithi002.stdout:68719476738 2024-05-08T15:59:55.067 INFO:teuthology.orchestra.run.smithi002.stdout:64424509442 2024-05-08T15:59:55.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476740 for osd.6 2024-05-08T15:59:55.079 INFO:teuthology.orchestra.run.smithi002.stdout:68719476739 2024-05-08T15:59:55.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476738 for osd.7 2024-05-08T15:59:55.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509442 for osd.3 2024-05-08T15:59:55.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476739 for osd.5 2024-05-08T15:59:55.128 INFO:teuthology.orchestra.run.smithi002.stdout:64424509443 2024-05-08T15:59:55.140 INFO:teuthology.orchestra.run.smithi002.stdout:64424509443 2024-05-08T15:59:55.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.2 2024-05-08T15:59:55.156 INFO:teuthology.orchestra.run.smithi002.stdout:64424509442 2024-05-08T15:59:55.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.0 2024-05-08T15:59:55.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509442 for osd.8 2024-05-08T15:59:55.177 INFO:teuthology.orchestra.run.smithi002.stdout:68719476739 2024-05-08T15:59:55.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476739 for osd.9 2024-05-08T15:59:55.233 INFO:teuthology.orchestra.run.smithi002.stdout:64424509442 2024-05-08T15:59:55.241 INFO:teuthology.orchestra.run.smithi002.stdout:64424509443 2024-05-08T15:59:55.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509442 for osd.4 2024-05-08T15:59:55.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509443 for osd.1 2024-05-08T15:59:56.077 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-08T15:59:56.101 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-08T15:59:56.112 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-08T15:59:56.115 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-08T15:59:56.148 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-08T15:59:56.167 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-08T15:59:56.175 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-05-08T15:59:56.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-05-08T15:59:56.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-08T15:59:56.260 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-08T15:59:56.655 INFO:teuthology.orchestra.run.smithi002.stdout:68719476740 2024-05-08T15:59:56.678 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-05-08T15:59:56.681 INFO:teuthology.orchestra.run.smithi002.stdout:68719476742 2024-05-08T15:59:56.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476739 got 68719476740 for osd.7 2024-05-08T15:59:56.697 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.708 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476742 for osd.6 2024-05-08T15:59:56.708 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476741 for osd.5 2024-05-08T15:59:56.721 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.760 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:56.775 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:56.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.8 2024-05-08T15:59:56.778 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.791 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:56.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.3 2024-05-08T15:59:56.794 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.0 2024-05-08T15:59:56.815 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.815 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:56.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.1 2024-05-08T15:59:56.833 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.839 INFO:teuthology.orchestra.run.smithi002.stdout:68719476741 2024-05-08T15:59:56.842 INFO:teuthology.orchestra.run.smithi002.stdout:64424509445 2024-05-08T15:59:56.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476740 got 68719476741 for osd.9 2024-05-08T15:59:56.859 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.2 2024-05-08T15:59:56.861 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.875 INFO:teuthology.orchestra.run.smithi002.stdout:64424509444 2024-05-08T15:59:56.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.4 2024-05-08T15:59:56.892 DEBUG:teuthology.parallel:result is None 2024-05-08T15:59:56.892 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-08T15:59:56.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T15:59:57.215 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:57.216 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T15:59:57.233 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-05-08T15:59:56.042384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":284512,"kb_used_data":1784,"kb_used_omap":15,"kb_used_meta":282544,"kb_avail":1048291488,"statfs":{"total":1073741824000,"available":1073450483712,"internally_reserved":0,"allocated":1826816,"data_stored":588912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":289325493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.224683+0000","last_change":"2024-05-08T15:59:53.181598+0000","last_active":"2024-05-08T15:59:54.224683+0000","last_peered":"2024-05-08T15:59:54.224683+0000","last_clean":"2024-05-08T15:59:54.224683+0000","last_became_active":"2024-05-08T15:59:53.181312+0000","last_became_peered":"2024-05-08T15:59:53.181312+0000","last_unstale":"2024-05-08T15:59:54.224683+0000","last_undegraded":"2024-05-08T15:59:54.224683+0000","last_fullsized":"2024-05-08T15:59:54.224683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T19:52:00.629662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.224682+0000","last_change":"2024-05-08T15:59:53.179900+0000","last_active":"2024-05-08T15:59:54.224682+0000","last_peered":"2024-05-08T15:59:54.224682+0000","last_clean":"2024-05-08T15:59:54.224682+0000","last_became_active":"2024-05-08T15:59:53.179527+0000","last_became_peered":"2024-05-08T15:59:53.179527+0000","last_unstale":"2024-05-08T15:59:54.224682+0000","last_undegraded":"2024-05-08T15:59:54.224682+0000","last_fullsized":"2024-05-08T15:59:54.224682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:36:22.893296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.159621+0000","last_change":"2024-05-08T15:59:53.158154+0000","last_active":"2024-05-08T15:59:53.159621+0000","last_peered":"2024-05-08T15:59:53.159621+0000","last_clean":"2024-05-08T15:59:53.159621+0000","last_became_active":"2024-05-08T15:59:53.157772+0000","last_became_peered":"2024-05-08T15:59:53.157772+0000","last_unstale":"2024-05-08T15:59:53.159621+0000","last_undegraded":"2024-05-08T15:59:53.159621+0000","last_fullsized":"2024-05-08T15:59:53.159621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:33:46.761991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.159942+0000","last_change":"2024-05-08T15:59:53.159269+0000","last_active":"2024-05-08T15:59:53.159942+0000","last_peered":"2024-05-08T15:59:53.159942+0000","last_clean":"2024-05-08T15:59:53.159942+0000","last_became_active":"2024-05-08T15:59:53.158605+0000","last_became_peered":"2024-05-08T15:59:53.158605+0000","last_unstale":"2024-05-08T15:59:53.159942+0000","last_undegraded":"2024-05-08T15:59:53.159942+0000","last_fullsized":"2024-05-08T15:59:53.159942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:29:41.092238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.180351+0000","last_change":"2024-05-08T15:59:53.179622+0000","last_active":"2024-05-08T15:59:53.180351+0000","last_peered":"2024-05-08T15:59:53.180351+0000","last_clean":"2024-05-08T15:59:53.180351+0000","last_became_active":"2024-05-08T15:59:53.179116+0000","last_became_peered":"2024-05-08T15:59:53.179116+0000","last_unstale":"2024-05-08T15:59:53.180351+0000","last_undegraded":"2024-05-08T15:59:53.180351+0000","last_fullsized":"2024-05-08T15:59:53.180351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:27:24.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.181176+0000","last_change":"2024-05-08T15:59:53.180180+0000","last_active":"2024-05-08T15:59:53.181176+0000","last_peered":"2024-05-08T15:59:53.181176+0000","last_clean":"2024-05-08T15:59:53.181176+0000","last_became_active":"2024-05-08T15:59:53.179766+0000","last_became_peered":"2024-05-08T15:59:53.179766+0000","last_unstale":"2024-05-08T15:59:53.181176+0000","last_undegraded":"2024-05-08T15:59:53.181176+0000","last_fullsized":"2024-05-08T15:59:53.181176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:08:33.560232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.157839+0000","last_change":"2024-05-08T15:59:53.157965+0000","last_active":"2024-05-08T15:59:54.157839+0000","last_peered":"2024-05-08T15:59:54.157839+0000","last_clean":"2024-05-08T15:59:54.157839+0000","last_became_active":"2024-05-08T15:59:53.157605+0000","last_became_peered":"2024-05-08T15:59:53.157605+0000","last_unstale":"2024-05-08T15:59:54.157839+0000","last_undegraded":"2024-05-08T15:59:54.157839+0000","last_fullsized":"2024-05-08T15:59:54.157839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:16:13.813710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.158729+0000","last_change":"2024-05-08T15:59:53.157871+0000","last_active":"2024-05-08T15:59:53.158729+0000","last_peered":"2024-05-08T15:59:53.158729+0000","last_clean":"2024-05-08T15:59:53.158729+0000","last_became_active":"2024-05-08T15:59:53.157444+0000","last_became_peered":"2024-05-08T15:59:53.157444+0000","last_unstale":"2024-05-08T15:59:53.158729+0000","last_undegraded":"2024-05-08T15:59:53.158729+0000","last_fullsized":"2024-05-08T15:59:53.158729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:33:46.761991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826240,"statfs":{"total":107374182400,"available":107342069760,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826240,"statfs":{"total":107374182400,"available":107342069760,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":192512,"data_stored":63784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31376,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826224,"statfs":{"total":107374182400,"available":107342053376,"internally_reserved":0,"allocated":192512,"data_stored":63784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T15:59:57.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T15:59:57.523 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:57.523 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T15:59:57.540 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-05-08T15:59:56.042384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":284512,"kb_used_data":1784,"kb_used_omap":15,"kb_used_meta":282544,"kb_avail":1048291488,"statfs":{"total":1073741824000,"available":1073450483712,"internally_reserved":0,"allocated":1826816,"data_stored":588912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":289325493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.224683+0000","last_change":"2024-05-08T15:59:53.181598+0000","last_active":"2024-05-08T15:59:54.224683+0000","last_peered":"2024-05-08T15:59:54.224683+0000","last_clean":"2024-05-08T15:59:54.224683+0000","last_became_active":"2024-05-08T15:59:53.181312+0000","last_became_peered":"2024-05-08T15:59:53.181312+0000","last_unstale":"2024-05-08T15:59:54.224683+0000","last_undegraded":"2024-05-08T15:59:54.224683+0000","last_fullsized":"2024-05-08T15:59:54.224683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T19:52:00.629662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.224682+0000","last_change":"2024-05-08T15:59:53.179900+0000","last_active":"2024-05-08T15:59:54.224682+0000","last_peered":"2024-05-08T15:59:54.224682+0000","last_clean":"2024-05-08T15:59:54.224682+0000","last_became_active":"2024-05-08T15:59:53.179527+0000","last_became_peered":"2024-05-08T15:59:53.179527+0000","last_unstale":"2024-05-08T15:59:54.224682+0000","last_undegraded":"2024-05-08T15:59:54.224682+0000","last_fullsized":"2024-05-08T15:59:54.224682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:36:22.893296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.159621+0000","last_change":"2024-05-08T15:59:53.158154+0000","last_active":"2024-05-08T15:59:53.159621+0000","last_peered":"2024-05-08T15:59:53.159621+0000","last_clean":"2024-05-08T15:59:53.159621+0000","last_became_active":"2024-05-08T15:59:53.157772+0000","last_became_peered":"2024-05-08T15:59:53.157772+0000","last_unstale":"2024-05-08T15:59:53.159621+0000","last_undegraded":"2024-05-08T15:59:53.159621+0000","last_fullsized":"2024-05-08T15:59:53.159621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:33:46.761991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.159942+0000","last_change":"2024-05-08T15:59:53.159269+0000","last_active":"2024-05-08T15:59:53.159942+0000","last_peered":"2024-05-08T15:59:53.159942+0000","last_clean":"2024-05-08T15:59:53.159942+0000","last_became_active":"2024-05-08T15:59:53.158605+0000","last_became_peered":"2024-05-08T15:59:53.158605+0000","last_unstale":"2024-05-08T15:59:53.159942+0000","last_undegraded":"2024-05-08T15:59:53.159942+0000","last_fullsized":"2024-05-08T15:59:53.159942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:29:41.092238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.180351+0000","last_change":"2024-05-08T15:59:53.179622+0000","last_active":"2024-05-08T15:59:53.180351+0000","last_peered":"2024-05-08T15:59:53.180351+0000","last_clean":"2024-05-08T15:59:53.180351+0000","last_became_active":"2024-05-08T15:59:53.179116+0000","last_became_peered":"2024-05-08T15:59:53.179116+0000","last_unstale":"2024-05-08T15:59:53.180351+0000","last_undegraded":"2024-05-08T15:59:53.180351+0000","last_fullsized":"2024-05-08T15:59:53.180351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:27:24.025845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.181176+0000","last_change":"2024-05-08T15:59:53.180180+0000","last_active":"2024-05-08T15:59:53.181176+0000","last_peered":"2024-05-08T15:59:53.181176+0000","last_clean":"2024-05-08T15:59:53.181176+0000","last_became_active":"2024-05-08T15:59:53.179766+0000","last_became_peered":"2024-05-08T15:59:53.179766+0000","last_unstale":"2024-05-08T15:59:53.181176+0000","last_undegraded":"2024-05-08T15:59:53.181176+0000","last_fullsized":"2024-05-08T15:59:53.181176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:08:33.560232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-08T15:59:54.157839+0000","last_change":"2024-05-08T15:59:53.157965+0000","last_active":"2024-05-08T15:59:54.157839+0000","last_peered":"2024-05-08T15:59:54.157839+0000","last_clean":"2024-05-08T15:59:54.157839+0000","last_became_active":"2024-05-08T15:59:53.157605+0000","last_became_peered":"2024-05-08T15:59:53.157605+0000","last_unstale":"2024-05-08T15:59:54.157839+0000","last_undegraded":"2024-05-08T15:59:54.157839+0000","last_fullsized":"2024-05-08T15:59:54.157839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:16:13.813710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-08T15:59:53.158729+0000","last_change":"2024-05-08T15:59:53.157871+0000","last_active":"2024-05-08T15:59:53.158729+0000","last_peered":"2024-05-08T15:59:53.158729+0000","last_clean":"2024-05-08T15:59:53.158729+0000","last_became_active":"2024-05-08T15:59:53.157444+0000","last_became_peered":"2024-05-08T15:59:53.157444+0000","last_unstale":"2024-05-08T15:59:53.158729+0000","last_undegraded":"2024-05-08T15:59:53.158729+0000","last_fullsized":"2024-05-08T15:59:53.158729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:33:46.761991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":16,"seq":68719476741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826240,"statfs":{"total":107374182400,"available":107342069760,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":16,"seq":68719476740,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31360,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826240,"statfs":{"total":107374182400,"available":107342069760,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":16,"seq":68719476742,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830392,"statfs":{"total":107374182400,"available":107346321408,"internally_reserved":0,"allocated":192512,"data_stored":63784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":16,"seq":68719476741,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31376,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104826224,"statfs":{"total":107374182400,"available":107342053376,"internally_reserved":0,"allocated":192512,"data_stored":63784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":15,"seq":64424509444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509445,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830400,"statfs":{"total":107374182400,"available":107346329600,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830396,"statfs":{"total":107374182400,"available":107346325504,"internally_reserved":0,"allocated":180224,"data_stored":57668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T15:59:57.541 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-08T15:59:57.541 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-08T15:59:57.541 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-08T15:59:57.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-08T15:59:57.852 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T15:59:57.852 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-08T15:59:57.868 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-08T15:59:57.869 INFO:teuthology.run_tasks:Running task workunit... 2024-05-08T15:59:57.883 INFO:tasks.workunit:Pulling workunits from ref 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:59:57.883 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-08T15:59:57.883 INFO:tasks.workunit:timeout=3h 2024-05-08T15:59:57.883 INFO:tasks.workunit:cleanup=True 2024-05-08T15:59:57.883 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-08T15:59:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:59:57.889 INFO:teuthology.orchestra.run.smithi002.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-08T15:59:57.889 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T15:59:57.935 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-08T15:59:57.935 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-08T15:59:57.983 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:59:58.034 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-08T16:01:36.288 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 95% (11698/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '95a4ad3c90f2f12f920e87725ad8d9002f948722'. 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.802 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-05-08T16:01:36.803 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 95a4ad3c90f Merge branch 'wip-rf-reinstate-q-squid' of https://github.com/ronen-fr/ceph into wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T16:01:36.809 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-08T16:01:36.859 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-08T16:01:36.862 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-08T16:01:36.863 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-08T16:01:37.634 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-08T16:01:37.713 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-08T16:01:37.767 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-08T16:01:37.768 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-08T16:01:37.768 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-08T16:01:37.816 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-08T16:01:37.819 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-05-08T16:01:37.819 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-08T16:01:37.870 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2024-05-08T16:01:37.871 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2024-05-08T16:01:37.871 DEBUG:teuthology.orchestra.run.smithi002:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a4ad3c90f2f12f920e87725ad8d9002f948722 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2024-05-08T16:01:37.923 INFO:tasks.workunit.client.0.smithi002.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2024-05-08T16:01:37.944 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd setcrushmap -i crushmap 2024-05-08T16:01:40.240 INFO:tasks.workunit.client.0.smithi002.stderr:6 2024-05-08T16:01:40.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd tree 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout:-11 10.00000 root root 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -9 8.00000 row row0 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -6 4.00000 rack rack0 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -1 2.00000 host host0 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -2 2.00000 host host1 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -7 4.00000 rack rack1 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: -3 2.00000 host host2 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi002.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: -4 2.00000 host host3 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout:-10 2.00000 row row1 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: -8 2.00000 rack rack2 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: -5 2.00000 host host4 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2024-05-08T16:01:40.674 INFO:tasks.workunit.client.0.smithi002.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2024-05-08T16:01:40.691 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2024-05-08T16:01:40.851 INFO:tasks.workunit.client.0.smithi002.stdout:osd.0: {} 2024-05-08T16:01:40.884 INFO:tasks.workunit.client.0.smithi002.stdout:osd.1: {} 2024-05-08T16:01:40.920 INFO:tasks.workunit.client.0.smithi002.stdout:osd.2: {} 2024-05-08T16:01:40.961 INFO:tasks.workunit.client.0.smithi002.stdout:osd.3: {} 2024-05-08T16:01:40.996 INFO:tasks.workunit.client.0.smithi002.stdout:osd.4: {} 2024-05-08T16:01:41.036 INFO:tasks.workunit.client.0.smithi002.stdout:osd.5: {} 2024-05-08T16:01:41.074 INFO:tasks.workunit.client.0.smithi002.stdout:osd.6: {} 2024-05-08T16:01:41.108 INFO:tasks.workunit.client.0.smithi002.stdout:osd.7: {} 2024-05-08T16:01:41.146 INFO:tasks.workunit.client.0.smithi002.stdout:osd.8: {} 2024-05-08T16:01:41.179 INFO:tasks.workunit.client.0.smithi002.stdout:osd.9: {} 2024-05-08T16:01:41.197 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noout 2024-05-08T16:01:43.261 INFO:tasks.workunit.client.0.smithi002.stderr:noout is set 2024-05-08T16:01:43.283 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_host_down 2024-05-08T16:01:43.284 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-05-08T16:01:45.273 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-05-08T16:01:45.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.0 osd.1 2024-05-08T16:01:45.609 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:01:45.611+0000 7f74abfda640 -1 mon.a@0(leader).osd e26 definitely_dead 0 2024-05-08T16:01:46.383 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:01:46.387+0000 7f74abfda640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2024-05-08T16:01:46.383 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. 2024-05-08T16:01:46.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:01:46.907 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-05-08T16:01:46.908 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-05-08T16:01:46.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:46.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-05-08T16:01:47.395 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:01:47.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:47.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-05-08T16:01:47.863 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:01:47.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:47.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.0 2024-05-08T16:01:48.355 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:01:48.359+0000 7f3e61655640 -1 osd.1 29 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:48.413 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:01:48.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:48.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.1 2024-05-08T16:01:48.938 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:01:48.938 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-05-08T16:01:49.295 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:01:49.299+0000 7f3e5cc80640 -1 osd.1 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:01:50.298 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-05-08T16:01:50.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-05-08T16:01:50.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:50.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:01:50.749 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T16:01:50.751+0000 7fa37c0c3640 -1 osd.0 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:01:50.786 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noout flag(s) set; 1 osds down 2024-05-08T16:01:50.786 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 1 2024-05-08T16:01:51.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:51.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:01:52.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_rack_down 2024-05-08T16:01:52.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-05-08T16:01:54.322 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-05-08T16:01:54.349 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.8 osd.9 2024-05-08T16:01:54.660 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:01:54.663+0000 7f74abfda640 -1 mon.a@0(leader).osd e35 definitely_dead 0 2024-05-08T16:01:55.435 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:01:55.439+0000 7f74abfda640 -1 mon.a@0(leader).osd e36 definitely_dead 0 2024-05-08T16:01:55.435 INFO:tasks.workunit.client.0.smithi002.stderr:osd.8 is already down. osd.9 is already down. 2024-05-08T16:01:55.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:55.802 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:01:55.807+0000 7fc82f598640 -1 osd.8 36 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:55.981 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2024-05-08T16:01:55.982 INFO:tasks.workunit.client.0.smithi002.stdout: row row1 (root=root) (2 osds) is down 2024-05-08T16:01:56.000 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:56.000 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-05-08T16:01:56.333 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:01:56.335+0000 7fc822ba2640 -1 osd.8 37 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:56.487 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-05-08T16:01:56.487 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:56.487 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 rack' 2024-05-08T16:01:56.957 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-05-08T16:01:56.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:56.957 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 row' 2024-05-08T16:01:57.109 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T16:01:57.115+0000 7fe124949640 -1 osd.9 37 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:57.345 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:01:57.347+0000 7fc822ba2640 -1 osd.8 38 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:57.345 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T16:01:57.347+0000 7fe11475d640 -1 osd.9 38 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:01:57.433 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-05-08T16:01:57.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:01:57.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-05-08T16:01:57.907 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2024-05-08T16:01:57.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:57.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.8 2024-05-08T16:01:58.419 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-05-08T16:01:58.420 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:01:58.420 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.9 2024-05-08T16:01:58.948 INFO:tasks.workunit.client.0.smithi002.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2024-05-08T16:01:58.949 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-05-08T16:01:59.352 INFO:tasks.ceph.osd.9.smithi002.stderr:2024-05-08T16:01:59.355+0000 7fe11c77e640 -1 osd.9 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:01:59.352 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:01:59.355+0000 7fc82abc3640 -1 osd.8 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:00.354 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-05-08T16:02:00.380 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-05-08T16:02:00.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:00.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:02:00.863 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_all_but_last_osds_down 2024-05-08T16:02:00.863 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-05-08T16:02:02.373 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-05-08T16:02:02.395 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph osd ls 2024-05-08T16:02:02.396 INFO:tasks.workunit.client.0.smithi002.stderr:++ sed '$d' 2024-05-08T16:02:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2024-05-08T16:02:03.134 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:02:03.139+0000 7f74abfda640 -1 mon.a@0(leader).osd e42 definitely_dead 0 2024-05-08T16:02:03.492 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:02:03.495+0000 7f74abfda640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-05-08T16:02:03.492 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2024-05-08T16:02:03.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:02:03.865 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:02:03.867+0000 7fc82ed86640 -1 osd.8 43 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:03.976 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T16:02:03.979+0000 7f41f8413640 -1 osd.4 43 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:04.047 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-05-08T16:02:04.047 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-05-08T16:02:04.047 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 9 osds down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-05-08T16:02:04.048 INFO:tasks.workunit.client.0.smithi002.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2024-05-08T16:02:04.049 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2024-05-08T16:02:04.049 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2024-05-08T16:02:04.049 INFO:tasks.workunit.client.0.smithi002.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2024-05-08T16:02:04.049 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2024-05-08T16:02:04.049 INFO:tasks.workunit.client.0.smithi002.stdout: row row0 (root=root) (8 osds) is down 2024-05-08T16:02:04.055 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T16:02:04.059+0000 7f4f3f929640 -1 osd.3 43 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:04.066 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:04.066 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 row' 2024-05-08T16:02:04.386 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T16:02:04.391+0000 7f4f2ef2b640 -1 osd.3 44 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:04.386 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:02:04.391+0000 7fc822ba2640 -1 osd.8 44 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:04.387 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T16:02:04.391+0000 7f41eba1d640 -1 osd.4 44 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:04.545 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-05-08T16:02:04.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:04.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 racks' 2024-05-08T16:02:04.668 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T16:02:04.671+0000 7f9851d7b640 -1 osd.7 44 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:05.034 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-05-08T16:02:05.035 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:05.035 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '4 hosts' 2024-05-08T16:02:05.392 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T16:02:05.395+0000 7f4f2ef2b640 -1 osd.3 45 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:05.392 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T16:02:05.395+0000 7f41eba1d640 -1 osd.4 45 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:05.392 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:02:05.395+0000 7fc822ba2640 -1 osd.8 45 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:05.392 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T16:02:05.395+0000 7f9841b8f640 -1 osd.7 45 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:05.517 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-05-08T16:02:05.517 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:05.517 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '9 osds' 2024-05-08T16:02:06.006 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2024-05-08T16:02:06.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-05-08T16:02:06.301 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T16:02:06.307+0000 7f77356f3640 -1 osd.6 45 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:06.395 INFO:tasks.ceph.osd.7.smithi002.stderr:2024-05-08T16:02:06.399+0000 7f9849bb0640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:06.395 INFO:tasks.ceph.osd.6.smithi002.stderr:2024-05-08T16:02:06.399+0000 7f772cd16640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:06.397 INFO:tasks.ceph.osd.8.smithi002.stderr:2024-05-08T16:02:06.399+0000 7fc82abc3640 -1 osd.8 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:06.397 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-05-08T16:02:06.399+0000 7f4f36f4c640 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:06.397 INFO:tasks.ceph.osd.4.smithi002.stderr:2024-05-08T16:02:06.399+0000 7f41f3a3e640 -1 osd.4 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:06.642 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T16:02:06.647+0000 7fa37c0c3640 -1 osd.0 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:07.175 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:02:07.179+0000 7f3e5cc80640 -1 osd.1 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:07.393 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-05-08T16:02:07.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-05-08T16:02:07.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:02:07.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:07.924 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noout flag(s) set; 2 osds down 2024-05-08T16:02:07.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 1 2024-05-08T16:02:08.411 INFO:tasks.ceph.osd.5.smithi002.stderr:2024-05-08T16:02:08.415+0000 7f5c57a02640 -1 osd.5 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:08.609 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-05-08T16:02:08.615+0000 7fb2e27ab640 -1 osd.2 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:02:09.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ test_mark_two_osds_same_host_down_with_classes 2024-05-08T16:02:09.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd set noup 2024-05-08T16:02:11.434 INFO:tasks.workunit.client.0.smithi002.stderr:noup is set 2024-05-08T16:02:11.457 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2024-05-08T16:02:13.440 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-05-08T16:02:13.463 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2024-05-08T16:02:15.452 INFO:tasks.workunit.client.0.smithi002.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-05-08T16:02:15.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd down osd.0 osd.1 2024-05-08T16:02:15.787 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:02:15.791+0000 7f74abfda640 -1 mon.a@0(leader).osd e55 definitely_dead 0 2024-05-08T16:02:16.574 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-05-08T16:02:16.579+0000 7f74abfda640 -1 mon.a@0(leader).osd e56 definitely_dead 0 2024-05-08T16:02:16.574 INFO:tasks.workunit.client.0.smithi002.stderr:osd.0 is already down. osd.1 is already down. 2024-05-08T16:02:16.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:02:17.109 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:02:17.109 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2024-05-08T16:02:17.109 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_DOWN: 2 osds down 2024-05-08T16:02:17.110 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:17.110 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:17.110 INFO:tasks.workunit.client.0.smithi002.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2024-05-08T16:02:17.110 INFO:tasks.workunit.client.0.smithi002.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2024-05-08T16:02:17.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:17.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '1 host' 2024-05-08T16:02:17.347 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:02:17.351+0000 7f3e61655640 -1 osd.1 56 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:17.463 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:02:17.467+0000 7f3e54c5f640 -1 osd.1 57 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:17.600 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:02:17.600 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:17.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep '2 osds' 2024-05-08T16:02:18.077 INFO:tasks.workunit.client.0.smithi002.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2024-05-08T16:02:18.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:02:18.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.0 2024-05-08T16:02:18.472 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:02:18.475+0000 7f3e54c5f640 -1 osd.1 58 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:18.595 INFO:tasks.workunit.client.0.smithi002.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:18.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health detail 2024-05-08T16:02:18.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep osd.1 2024-05-08T16:02:18.602 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T16:02:18.607+0000 7fa38428e640 -1 osd.0 58 osdmap NOUP flag is set, waiting for it to clear 2024-05-08T16:02:19.125 INFO:tasks.workunit.client.0.smithi002.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2024-05-08T16:02:19.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd unset noup 2024-05-08T16:02:19.475 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-05-08T16:02:19.479+0000 7fa37c0c3640 -1 osd.0 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:19.477 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-05-08T16:02:19.479+0000 7f3e5cc80640 -1 osd.1 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-08T16:02:20.477 INFO:tasks.workunit.client.0.smithi002.stderr:noup is unset 2024-05-08T16:02:20.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ wait_for_healthy 2024-05-08T16:02:20.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph health 2024-05-08T16:02:20.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep down 2024-05-08T16:02:20.980 INFO:tasks.workunit.client.0.smithi002.stderr:+ exit 0 2024-05-08T16:02:20.981 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T16:02:20.981 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-08T16:02:21.037 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2024-05-08T16:02:21.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-08T16:02:21.366 DEBUG:teuthology.parallel:result is None 2024-05-08T16:02:21.366 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T16:02:21.378 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T16:02:21.379 DEBUG:teuthology.orchestra.run.smithi002:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T16:02:21.431 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T16:02:21.432 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-08T16:02:21.444 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-08T16:02:21.444 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:21.775 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:21.775 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:21.794 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-05-08T16:02:21.492249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293268,"kb_used_data":6392,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282732,"statfs":{"total":1073741824000,"available":1073441517568,"internally_reserved":0,"allocated":6545408,"data_stored":3106896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.031975+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:19.031975+0000","last_peered":"2024-05-08T16:02:19.031975+0000","last_clean":"2024-05-08T16:02:19.031975+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:19.031975+0000","last_undegraded":"2024-05-08T16:02:19.031975+0000","last_fullsized":"2024-05-08T16:02:19.031975+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:19.963730+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:19.963730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":97,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:18.479888+0000","last_change":"2024-05-08T16:02:18.479888+0000","last_active":"2024-05-08T16:02:18.479888+0000","last_peered":"2024-05-08T16:02:18.479888+0000","last_clean":"2024-05-08T16:02:15.463209+0000","last_became_active":"2024-05-08T16:02:17.476466+0000","last_became_peered":"2024-05-08T16:02:17.476466+0000","last_unstale":"2024-05-08T16:02:18.479888+0000","last_undegraded":"2024-05-08T16:02:18.479888+0000","last_fullsized":"2024-05-08T16:02:17.475711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":59,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:17.475934+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:02.996265+0000","last_became_active":"2024-05-08T16:02:17.475934+0000","last_became_peered":"2024-05-08T16:02:17.475934+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:17.475026+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031784+0000","last_change":"2024-05-08T16:02:17.475915+0000","last_active":"2024-05-08T16:02:19.031784+0000","last_peered":"2024-05-08T16:02:19.031784+0000","last_clean":"2024-05-08T16:02:16.472294+0000","last_became_active":"2024-05-08T16:02:17.475915+0000","last_became_peered":"2024-05-08T16:02:17.475915+0000","last_unstale":"2024-05-08T16:02:19.031784+0000","last_undegraded":"2024-05-08T16:02:19.031784+0000","last_fullsized":"2024-05-08T16:02:17.474657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031681+0000","last_change":"2024-05-08T16:02:17.474993+0000","last_active":"2024-05-08T16:02:19.031681+0000","last_peered":"2024-05-08T16:02:19.031681+0000","last_clean":"2024-05-08T16:02:16.055019+0000","last_became_active":"2024-05-08T16:02:17.474993+0000","last_became_peered":"2024-05-08T16:02:17.474993+0000","last_unstale":"2024-05-08T16:02:19.031681+0000","last_undegraded":"2024-05-08T16:02:19.031681+0000","last_fullsized":"2024-05-08T16:02:17.473830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:30:30.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032566+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:19.032566+0000","last_peered":"2024-05-08T16:02:19.032566+0000","last_clean":"2024-05-08T16:02:19.032566+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:19.032566+0000","last_undegraded":"2024-05-08T16:02:19.032566+0000","last_fullsized":"2024-05-08T16:02:19.032566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032903+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:19.032903+0000","last_peered":"2024-05-08T16:02:19.032903+0000","last_clean":"2024-05-08T16:02:19.032903+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:19.032903+0000","last_undegraded":"2024-05-08T16:02:19.032903+0000","last_fullsized":"2024-05-08T16:02:19.032903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27660,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":34,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27676,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829924,"statfs":{"total":107374182400,"available":107345842176,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:21.796 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:22.096 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:22.097 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:22.116 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-05-08T16:02:22.078780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293268,"kb_used_data":6392,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282732,"statfs":{"total":1073741824000,"available":1073441517568,"internally_reserved":0,"allocated":6545408,"data_stored":3106896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.031975+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:19.031975+0000","last_peered":"2024-05-08T16:02:19.031975+0000","last_clean":"2024-05-08T16:02:19.031975+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:19.031975+0000","last_undegraded":"2024-05-08T16:02:19.031975+0000","last_fullsized":"2024-05-08T16:02:19.031975+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:19.963730+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:19.963730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":97,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:18.479888+0000","last_change":"2024-05-08T16:02:18.479888+0000","last_active":"2024-05-08T16:02:18.479888+0000","last_peered":"2024-05-08T16:02:18.479888+0000","last_clean":"2024-05-08T16:02:15.463209+0000","last_became_active":"2024-05-08T16:02:17.476466+0000","last_became_peered":"2024-05-08T16:02:17.476466+0000","last_unstale":"2024-05-08T16:02:18.479888+0000","last_undegraded":"2024-05-08T16:02:18.479888+0000","last_fullsized":"2024-05-08T16:02:17.475711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":59,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:17.475934+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:02.996265+0000","last_became_active":"2024-05-08T16:02:17.475934+0000","last_became_peered":"2024-05-08T16:02:17.475934+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:17.475026+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031784+0000","last_change":"2024-05-08T16:02:17.475915+0000","last_active":"2024-05-08T16:02:19.031784+0000","last_peered":"2024-05-08T16:02:19.031784+0000","last_clean":"2024-05-08T16:02:16.472294+0000","last_became_active":"2024-05-08T16:02:17.475915+0000","last_became_peered":"2024-05-08T16:02:17.475915+0000","last_unstale":"2024-05-08T16:02:19.031784+0000","last_undegraded":"2024-05-08T16:02:19.031784+0000","last_fullsized":"2024-05-08T16:02:17.474657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031681+0000","last_change":"2024-05-08T16:02:17.474993+0000","last_active":"2024-05-08T16:02:19.031681+0000","last_peered":"2024-05-08T16:02:19.031681+0000","last_clean":"2024-05-08T16:02:16.055019+0000","last_became_active":"2024-05-08T16:02:17.474993+0000","last_became_peered":"2024-05-08T16:02:17.474993+0000","last_unstale":"2024-05-08T16:02:19.031681+0000","last_undegraded":"2024-05-08T16:02:19.031681+0000","last_fullsized":"2024-05-08T16:02:17.473830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:30:30.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032566+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:19.032566+0000","last_peered":"2024-05-08T16:02:19.032566+0000","last_clean":"2024-05-08T16:02:19.032566+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:19.032566+0000","last_undegraded":"2024-05-08T16:02:19.032566+0000","last_fullsized":"2024-05-08T16:02:19.032566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032903+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:19.032903+0000","last_peered":"2024-05-08T16:02:19.032903+0000","last_clean":"2024-05-08T16:02:19.032903+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:19.032903+0000","last_undegraded":"2024-05-08T16:02:19.032903+0000","last_fullsized":"2024-05-08T16:02:19.032903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27660,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":34,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27676,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829924,"statfs":{"total":107374182400,"available":107345842176,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:22.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:22.419 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:22.420 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:22.439 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-05-08T16:02:22.078780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293268,"kb_used_data":6392,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282732,"statfs":{"total":1073741824000,"available":1073441517568,"internally_reserved":0,"allocated":6545408,"data_stored":3106896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.031975+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:19.031975+0000","last_peered":"2024-05-08T16:02:19.031975+0000","last_clean":"2024-05-08T16:02:19.031975+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:19.031975+0000","last_undegraded":"2024-05-08T16:02:19.031975+0000","last_fullsized":"2024-05-08T16:02:19.031975+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:19.963730+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:19.963730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":97,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:18.479888+0000","last_change":"2024-05-08T16:02:18.479888+0000","last_active":"2024-05-08T16:02:18.479888+0000","last_peered":"2024-05-08T16:02:18.479888+0000","last_clean":"2024-05-08T16:02:15.463209+0000","last_became_active":"2024-05-08T16:02:17.476466+0000","last_became_peered":"2024-05-08T16:02:17.476466+0000","last_unstale":"2024-05-08T16:02:18.479888+0000","last_undegraded":"2024-05-08T16:02:18.479888+0000","last_fullsized":"2024-05-08T16:02:17.475711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":59,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:17.475934+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:02.996265+0000","last_became_active":"2024-05-08T16:02:17.475934+0000","last_became_peered":"2024-05-08T16:02:17.475934+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:17.475026+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031784+0000","last_change":"2024-05-08T16:02:17.475915+0000","last_active":"2024-05-08T16:02:19.031784+0000","last_peered":"2024-05-08T16:02:19.031784+0000","last_clean":"2024-05-08T16:02:16.472294+0000","last_became_active":"2024-05-08T16:02:17.475915+0000","last_became_peered":"2024-05-08T16:02:17.475915+0000","last_unstale":"2024-05-08T16:02:19.031784+0000","last_undegraded":"2024-05-08T16:02:19.031784+0000","last_fullsized":"2024-05-08T16:02:17.474657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031681+0000","last_change":"2024-05-08T16:02:17.474993+0000","last_active":"2024-05-08T16:02:19.031681+0000","last_peered":"2024-05-08T16:02:19.031681+0000","last_clean":"2024-05-08T16:02:16.055019+0000","last_became_active":"2024-05-08T16:02:17.474993+0000","last_became_peered":"2024-05-08T16:02:17.474993+0000","last_unstale":"2024-05-08T16:02:19.031681+0000","last_undegraded":"2024-05-08T16:02:19.031681+0000","last_fullsized":"2024-05-08T16:02:17.473830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:30:30.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032566+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:19.032566+0000","last_peered":"2024-05-08T16:02:19.032566+0000","last_clean":"2024-05-08T16:02:19.032566+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:19.032566+0000","last_undegraded":"2024-05-08T16:02:19.032566+0000","last_fullsized":"2024-05-08T16:02:19.032566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032903+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:19.032903+0000","last_peered":"2024-05-08T16:02:19.032903+0000","last_clean":"2024-05-08T16:02:19.032903+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:19.032903+0000","last_undegraded":"2024-05-08T16:02:19.032903+0000","last_fullsized":"2024-05-08T16:02:19.032903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27660,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":34,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27676,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829924,"statfs":{"total":107374182400,"available":107345842176,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 97, 'reported_epoch': 58, 'state': 'active+undersized', 'last_fresh': '2024-05-08T16:02:18.479888+0000', 'last_change': '2024-05-08T16:02:18.479888+0000', 'last_active': '2024-05-08T16:02:18.479888+0000', 'last_peered': '2024-05-08T16:02:18.479888+0000', 'last_clean': '2024-05-08T16:02:15.463209+0000', 'last_became_active': '2024-05-08T16:02:17.476466+0000', 'last_became_peered': '2024-05-08T16:02:17.476466+0000', 'last_unstale': '2024-05-08T16:02:18.479888+0000', 'last_undegraded': '2024-05-08T16:02:18.479888+0000', 'last_fullsized': '2024-05-08T16:02:17.475711+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_clean_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 59, 'reported_epoch': 59, 'state': 'active+undersized', 'last_fresh': '2024-05-08T16:02:19.963730+0000', 'last_change': '2024-05-08T16:02:17.475934+0000', 'last_active': '2024-05-08T16:02:19.963730+0000', 'last_peered': '2024-05-08T16:02:19.963730+0000', 'last_clean': '2024-05-08T16:02:02.996265+0000', 'last_became_active': '2024-05-08T16:02:17.475934+0000', 'last_became_peered': '2024-05-08T16:02:17.475934+0000', 'last_unstale': '2024-05-08T16:02:19.963730+0000', 'last_undegraded': '2024-05-08T16:02:19.963730+0000', 'last_fullsized': '2024-05-08T16:02:17.475026+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_clean_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "0'0", 'reported_seq': 84, 'reported_epoch': 58, 'state': 'active+undersized', 'last_fresh': '2024-05-08T16:02:19.031784+0000', 'last_change': '2024-05-08T16:02:17.475915+0000', 'last_active': '2024-05-08T16:02:19.031784+0000', 'last_peered': '2024-05-08T16:02:19.031784+0000', 'last_clean': '2024-05-08T16:02:16.472294+0000', 'last_became_active': '2024-05-08T16:02:17.475915+0000', 'last_became_peered': '2024-05-08T16:02:17.475915+0000', 'last_unstale': '2024-05-08T16:02:19.031784+0000', 'last_undegraded': '2024-05-08T16:02:19.031784+0000', 'last_fullsized': '2024-05-08T16:02:17.474657+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_clean_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-08T16:02:22.440 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2024-05-08T16:02:22.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "0'0", 'reported_seq': 86, 'reported_epoch': 58, 'state': 'active+undersized', 'last_fresh': '2024-05-08T16:02:19.031681+0000', 'last_change': '2024-05-08T16:02:17.474993+0000', 'last_active': '2024-05-08T16:02:19.031681+0000', 'last_peered': '2024-05-08T16:02:19.031681+0000', 'last_clean': '2024-05-08T16:02:16.055019+0000', 'last_became_active': '2024-05-08T16:02:17.474993+0000', 'last_became_peered': '2024-05-08T16:02:17.474993+0000', 'last_unstale': '2024-05-08T16:02:19.031681+0000', 'last_undegraded': '2024-05-08T16:02:19.031681+0000', 'last_fullsized': '2024-05-08T16:02:17.473830+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'last_clean_scrub_stamp': '2024-05-08T15:59:52.141052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-05-09T17:30:30.247386+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-08T16:02:22.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-08T16:02:22.803 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:22.804 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":163,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":10,"num_up_osds":10,"osd_up_since":1715184140,"num_in_osds":10,"osd_in_since":1715183982,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized","count":4}],"num_pgs":8,"num_pools":1,"num_objects":0,"data_bytes":0,"bytes_used":300306432,"bytes_avail":1073441517568,"bytes_total":1073741824000},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-05-08T16:00:52.058922+0000","services":{}},"progress_events":{}} 2024-05-08T16:02:22.822 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-05-08T16:02:22.823 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:23.127 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:23.128 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:23.145 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-05-08T16:02:22.078780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293268,"kb_used_data":6392,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282732,"statfs":{"total":1073741824000,"available":1073441517568,"internally_reserved":0,"allocated":6545408,"data_stored":3106896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.031975+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:19.031975+0000","last_peered":"2024-05-08T16:02:19.031975+0000","last_clean":"2024-05-08T16:02:19.031975+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:19.031975+0000","last_undegraded":"2024-05-08T16:02:19.031975+0000","last_fullsized":"2024-05-08T16:02:19.031975+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:19.963730+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:19.963730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":97,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:18.479888+0000","last_change":"2024-05-08T16:02:18.479888+0000","last_active":"2024-05-08T16:02:18.479888+0000","last_peered":"2024-05-08T16:02:18.479888+0000","last_clean":"2024-05-08T16:02:15.463209+0000","last_became_active":"2024-05-08T16:02:17.476466+0000","last_became_peered":"2024-05-08T16:02:17.476466+0000","last_unstale":"2024-05-08T16:02:18.479888+0000","last_undegraded":"2024-05-08T16:02:18.479888+0000","last_fullsized":"2024-05-08T16:02:17.475711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":59,"reported_epoch":59,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.963730+0000","last_change":"2024-05-08T16:02:17.475934+0000","last_active":"2024-05-08T16:02:19.963730+0000","last_peered":"2024-05-08T16:02:19.963730+0000","last_clean":"2024-05-08T16:02:02.996265+0000","last_became_active":"2024-05-08T16:02:17.475934+0000","last_became_peered":"2024-05-08T16:02:17.475934+0000","last_unstale":"2024-05-08T16:02:19.963730+0000","last_undegraded":"2024-05-08T16:02:19.963730+0000","last_fullsized":"2024-05-08T16:02:17.475026+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031784+0000","last_change":"2024-05-08T16:02:17.475915+0000","last_active":"2024-05-08T16:02:19.031784+0000","last_peered":"2024-05-08T16:02:19.031784+0000","last_clean":"2024-05-08T16:02:16.472294+0000","last_became_active":"2024-05-08T16:02:17.475915+0000","last_became_peered":"2024-05-08T16:02:17.475915+0000","last_unstale":"2024-05-08T16:02:19.031784+0000","last_undegraded":"2024-05-08T16:02:19.031784+0000","last_fullsized":"2024-05-08T16:02:17.474657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:00:52.667622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":86,"reported_epoch":58,"state":"active+undersized","last_fresh":"2024-05-08T16:02:19.031681+0000","last_change":"2024-05-08T16:02:17.474993+0000","last_active":"2024-05-08T16:02:19.031681+0000","last_peered":"2024-05-08T16:02:19.031681+0000","last_clean":"2024-05-08T16:02:16.055019+0000","last_became_active":"2024-05-08T16:02:17.474993+0000","last_became_peered":"2024-05-08T16:02:17.474993+0000","last_unstale":"2024-05-08T16:02:19.031681+0000","last_undegraded":"2024-05-08T16:02:19.031681+0000","last_fullsized":"2024-05-08T16:02:17.473830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:30:30.247386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032566+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:19.032566+0000","last_peered":"2024-05-08T16:02:19.032566+0000","last_clean":"2024-05-08T16:02:19.032566+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:19.032566+0000","last_undegraded":"2024-05-08T16:02:19.032566+0000","last_fullsized":"2024-05-08T16:02:19.032566+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-05-08T16:02:19.032903+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:19.032903+0000","last_peered":"2024-05-08T16:02:19.032903+0000","last_clean":"2024-05-08T16:02:19.032903+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:19.032903+0000","last_undegraded":"2024-05-08T16:02:19.032903+0000","last_fullsized":"2024-05-08T16:02:19.032903+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31824,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825776,"statfs":{"total":107374182400,"available":107341594624,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27660,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829940,"statfs":{"total":107374182400,"available":107345858560,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462945,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397538,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829944,"statfs":{"total":107374182400,"available":107345862656,"internally_reserved":0,"allocated":647168,"data_stored":306078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":34,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27676,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829924,"statfs":{"total":107374182400,"available":107345842176,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27668,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829932,"statfs":{"total":107374182400,"available":107345850368,"internally_reserved":0,"allocated":659456,"data_stored":313764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:26.147 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:26.456 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:26.456 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-05-08T16:02:26.079813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293676,"kb_used_data":6800,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282324,"statfs":{"total":1073741824000,"available":1073441099776,"internally_reserved":0,"allocated":6963200,"data_stored":3375440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.496154+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:21.496154+0000","last_peered":"2024-05-08T16:02:21.496154+0000","last_clean":"2024-05-08T16:02:21.496154+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:21.496154+0000","last_undegraded":"2024-05-08T16:02:21.496154+0000","last_fullsized":"2024-05-08T16:02:21.496154+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522752+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:21.522752+0000","last_peered":"2024-05-08T16:02:21.522752+0000","last_clean":"2024-05-08T16:02:21.522752+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:21.522752+0000","last_undegraded":"2024-05-08T16:02:21.522752+0000","last_fullsized":"2024-05-08T16:02:21.522752+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.524888+0000","last_change":"2024-05-08T16:02:21.524888+0000","last_active":"2024-05-08T16:02:21.524888+0000","last_peered":"2024-05-08T16:02:21.524888+0000","last_clean":"2024-05-08T16:02:21.524888+0000","last_became_active":"2024-05-08T16:02:21.524452+0000","last_became_peered":"2024-05-08T16:02:21.524452+0000","last_unstale":"2024-05-08T16:02:21.524888+0000","last_undegraded":"2024-05-08T16:02:21.524888+0000","last_fullsized":"2024-05-08T16:02:21.524888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:22:34.099380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.524959+0000","last_change":"2024-05-08T16:02:21.524959+0000","last_active":"2024-05-08T16:02:21.524959+0000","last_peered":"2024-05-08T16:02:21.524959+0000","last_clean":"2024-05-08T16:02:21.524959+0000","last_became_active":"2024-05-08T16:02:21.524534+0000","last_became_peered":"2024-05-08T16:02:21.524534+0000","last_unstale":"2024-05-08T16:02:21.524959+0000","last_undegraded":"2024-05-08T16:02:21.524959+0000","last_fullsized":"2024-05-08T16:02:21.524959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:12:40.583647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498540+0000","last_change":"2024-05-08T16:02:21.498540+0000","last_active":"2024-05-08T16:02:21.498540+0000","last_peered":"2024-05-08T16:02:21.498540+0000","last_clean":"2024-05-08T16:02:21.498540+0000","last_became_active":"2024-05-08T16:02:21.498129+0000","last_became_peered":"2024-05-08T16:02:21.498129+0000","last_unstale":"2024-05-08T16:02:21.498540+0000","last_undegraded":"2024-05-08T16:02:21.498540+0000","last_fullsized":"2024-05-08T16:02:21.498540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":97,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498390+0000","last_change":"2024-05-08T16:02:21.498390+0000","last_active":"2024-05-08T16:02:21.498390+0000","last_peered":"2024-05-08T16:02:21.498390+0000","last_clean":"2024-05-08T16:02:21.498390+0000","last_became_active":"2024-05-08T16:02:21.498079+0000","last_became_peered":"2024-05-08T16:02:21.498079+0000","last_unstale":"2024-05-08T16:02:21.498390+0000","last_undegraded":"2024-05-08T16:02:21.498390+0000","last_fullsized":"2024-05-08T16:02:21.498390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T18:16:41.125478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.523069+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:21.523069+0000","last_peered":"2024-05-08T16:02:21.523069+0000","last_clean":"2024-05-08T16:02:21.523069+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:21.523069+0000","last_undegraded":"2024-05-08T16:02:21.523069+0000","last_fullsized":"2024-05-08T16:02:21.523069+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522687+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:21.522687+0000","last_peered":"2024-05-08T16:02:21.522687+0000","last_clean":"2024-05-08T16:02:21.522687+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:21.522687+0000","last_undegraded":"2024-05-08T16:02:21.522687+0000","last_fullsized":"2024-05-08T16:02:21.522687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462948,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462946,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462946,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037795,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:26.475 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-08T16:02:26.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:26.781 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:26.781 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-05-08T16:02:26.079813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293676,"kb_used_data":6800,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282324,"statfs":{"total":1073741824000,"available":1073441099776,"internally_reserved":0,"allocated":6963200,"data_stored":3375440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.496154+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:21.496154+0000","last_peered":"2024-05-08T16:02:21.496154+0000","last_clean":"2024-05-08T16:02:21.496154+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:21.496154+0000","last_undegraded":"2024-05-08T16:02:21.496154+0000","last_fullsized":"2024-05-08T16:02:21.496154+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522752+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:21.522752+0000","last_peered":"2024-05-08T16:02:21.522752+0000","last_clean":"2024-05-08T16:02:21.522752+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:21.522752+0000","last_undegraded":"2024-05-08T16:02:21.522752+0000","last_fullsized":"2024-05-08T16:02:21.522752+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.524888+0000","last_change":"2024-05-08T16:02:21.524888+0000","last_active":"2024-05-08T16:02:21.524888+0000","last_peered":"2024-05-08T16:02:21.524888+0000","last_clean":"2024-05-08T16:02:21.524888+0000","last_became_active":"2024-05-08T16:02:21.524452+0000","last_became_peered":"2024-05-08T16:02:21.524452+0000","last_unstale":"2024-05-08T16:02:21.524888+0000","last_undegraded":"2024-05-08T16:02:21.524888+0000","last_fullsized":"2024-05-08T16:02:21.524888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:22:34.099380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.524959+0000","last_change":"2024-05-08T16:02:21.524959+0000","last_active":"2024-05-08T16:02:21.524959+0000","last_peered":"2024-05-08T16:02:21.524959+0000","last_clean":"2024-05-08T16:02:21.524959+0000","last_became_active":"2024-05-08T16:02:21.524534+0000","last_became_peered":"2024-05-08T16:02:21.524534+0000","last_unstale":"2024-05-08T16:02:21.524959+0000","last_undegraded":"2024-05-08T16:02:21.524959+0000","last_fullsized":"2024-05-08T16:02:21.524959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:12:40.583647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498540+0000","last_change":"2024-05-08T16:02:21.498540+0000","last_active":"2024-05-08T16:02:21.498540+0000","last_peered":"2024-05-08T16:02:21.498540+0000","last_clean":"2024-05-08T16:02:21.498540+0000","last_became_active":"2024-05-08T16:02:21.498129+0000","last_became_peered":"2024-05-08T16:02:21.498129+0000","last_unstale":"2024-05-08T16:02:21.498540+0000","last_undegraded":"2024-05-08T16:02:21.498540+0000","last_fullsized":"2024-05-08T16:02:21.498540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":97,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498390+0000","last_change":"2024-05-08T16:02:21.498390+0000","last_active":"2024-05-08T16:02:21.498390+0000","last_peered":"2024-05-08T16:02:21.498390+0000","last_clean":"2024-05-08T16:02:21.498390+0000","last_became_active":"2024-05-08T16:02:21.498079+0000","last_became_peered":"2024-05-08T16:02:21.498079+0000","last_unstale":"2024-05-08T16:02:21.498390+0000","last_undegraded":"2024-05-08T16:02:21.498390+0000","last_fullsized":"2024-05-08T16:02:21.498390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T18:16:41.125478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.523069+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:21.523069+0000","last_peered":"2024-05-08T16:02:21.523069+0000","last_clean":"2024-05-08T16:02:21.523069+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:21.523069+0000","last_undegraded":"2024-05-08T16:02:21.523069+0000","last_fullsized":"2024-05-08T16:02:21.523069+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522687+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:21.522687+0000","last_peered":"2024-05-08T16:02:21.522687+0000","last_clean":"2024-05-08T16:02:21.522687+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:21.522687+0000","last_undegraded":"2024-05-08T16:02:21.522687+0000","last_fullsized":"2024-05-08T16:02:21.522687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462948,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462946,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462946,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397539,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037795,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:26.800 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-08T16:02:27.104 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:27.104 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":61,"fsid":"84e991d3-12d8-4331-8afe-a495890a074b","created":"2024-05-08T15:59:38.986787+0000","modified":"2024-05-08T16:02:21.490389+0000","last_up_change":"2024-05-08T16:02:20.483294+0000","last_in_change":"2024-05-08T15:59:42.114535+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-08T15:59:51.300852+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.80000001192092896,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1.0000001192092896,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2253cfc3-2eed-40f9-b09a-ee5ed2d98ce3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":59,"up_from":60,"up_thru":36,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6841","nonce":2047446632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2050446632},{"type":"v1","addr":"172.21.15.2:6811","nonce":2050446632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6846","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6847","nonce":2047446632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6844","nonce":2047446632},{"type":"v1","addr":"172.21.15.2:6845","nonce":2047446632}]},"public_addr":"172.21.15.2:6841/2047446632","cluster_addr":"172.21.15.2:6811/2050446632","heartbeat_back_addr":"172.21.15.2:6847/2047446632","heartbeat_front_addr":"172.21.15.2:6845/2047446632","state":["exists","up"]},{"osd":1,"uuid":"bccff570-697e-41bb-a25e-485c4f834c2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":59,"up_from":60,"up_thru":60,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6809","nonce":3509145648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6856","nonce":3512145648},{"type":"v1","addr":"172.21.15.2:6861","nonce":3512145648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6815","nonce":3509145648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":3509145648},{"type":"v1","addr":"172.21.15.2:6813","nonce":3509145648}]},"public_addr":"172.21.15.2:6809/3509145648","cluster_addr":"172.21.15.2:6861/3512145648","heartbeat_back_addr":"172.21.15.2:6815/3509145648","heartbeat_front_addr":"172.21.15.2:6813/3509145648","state":["exists","up"]},{"osd":2,"uuid":"0c855c70-a738-4ecd-acd2-7df525160975","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":48,"up_from":49,"up_thru":0,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6801","nonce":4107948721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6882","nonce":4108948721},{"type":"v1","addr":"172.21.15.2:6883","nonce":4108948721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6807","nonce":4107948721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":4107948721},{"type":"v1","addr":"172.21.15.2:6805","nonce":4107948721}]},"public_addr":"172.21.15.2:6801/4107948721","cluster_addr":"172.21.15.2:6883/4108948721","heartbeat_back_addr":"172.21.15.2:6807/4107948721","heartbeat_front_addr":"172.21.15.2:6805/4107948721","state":["exists","up"]},{"osd":3,"uuid":"54600c26-8a38-47dc-932a-d17e98b047cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":60,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6821","nonce":607357490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6842","nonce":608357490},{"type":"v1","addr":"172.21.15.2:6843","nonce":608357490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6831","nonce":607357490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":607357490},{"type":"v1","addr":"172.21.15.2:6829","nonce":607357490}]},"public_addr":"172.21.15.2:6821/607357490","cluster_addr":"172.21.15.2:6843/608357490","heartbeat_back_addr":"172.21.15.2:6831/607357490","heartbeat_front_addr":"172.21.15.2:6829/607357490","state":["exists","up"]},{"osd":4,"uuid":"ddca8acb-7e0e-4768-9b4c-3b1fecb9c240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":60,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6817","nonce":3384451429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3385451429},{"type":"v1","addr":"172.21.15.2:6862","nonce":3385451429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6826","nonce":3384451429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":3384451429},{"type":"v1","addr":"172.21.15.2:6823","nonce":3384451429}]},"public_addr":"172.21.15.2:6817/3384451429","cluster_addr":"172.21.15.2:6862/3385451429","heartbeat_back_addr":"172.21.15.2:6826/3384451429","heartbeat_front_addr":"172.21.15.2:6823/3384451429","state":["exists","up"]},{"osd":5,"uuid":"7a051df4-19a4-4c95-8569-74438658105d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":48,"up_from":49,"up_thru":49,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6848","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6852","nonce":364873758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":365873758},{"type":"v1","addr":"172.21.15.2:6803","nonce":365873758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6872","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6876","nonce":364873758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6864","nonce":364873758},{"type":"v1","addr":"172.21.15.2:6868","nonce":364873758}]},"public_addr":"172.21.15.2:6852/364873758","cluster_addr":"172.21.15.2:6803/365873758","heartbeat_back_addr":"172.21.15.2:6876/364873758","heartbeat_front_addr":"172.21.15.2:6868/364873758","state":["exists","up"]},{"osd":6,"uuid":"ae09d8e2-752a-4b16-a1ed-6ad50ec7f4ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":56,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6833","nonce":2323196745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6857","nonce":2324196745},{"type":"v1","addr":"172.21.15.2:6860","nonce":2324196745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6839","nonce":2323196745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2323196745},{"type":"v1","addr":"172.21.15.2:6837","nonce":2323196745}]},"public_addr":"172.21.15.2:6833/2323196745","cluster_addr":"172.21.15.2:6860/2324196745","heartbeat_back_addr":"172.21.15.2:6839/2323196745","heartbeat_front_addr":"172.21.15.2:6837/2323196745","state":["exists","up"]},{"osd":7,"uuid":"448ff819-e3da-47db-8423-873815a23025","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6850","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6854","nonce":672606001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6819","nonce":673606001},{"type":"v1","addr":"172.21.15.2:6820","nonce":673606001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6873","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6877","nonce":672606001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6865","nonce":672606001},{"type":"v1","addr":"172.21.15.2:6869","nonce":672606001}]},"public_addr":"172.21.15.2:6854/672606001","cluster_addr":"172.21.15.2:6820/673606001","heartbeat_back_addr":"172.21.15.2:6877/672606001","heartbeat_front_addr":"172.21.15.2:6869/672606001","state":["exists","up"]},{"osd":8,"uuid":"908caf6f-0199-481f-8fef-810c6bbec8ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":18,"down_at":43,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6851","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6855","nonce":1901204544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6858","nonce":1903204544},{"type":"v1","addr":"172.21.15.2:6824","nonce":1903204544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6875","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6879","nonce":1901204544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6867","nonce":1901204544},{"type":"v1","addr":"172.21.15.2:6871","nonce":1901204544}]},"public_addr":"172.21.15.2:6855/1901204544","cluster_addr":"172.21.15.2:6824/1903204544","heartbeat_back_addr":"172.21.15.2:6879/1901204544","heartbeat_front_addr":"172.21.15.2:6871/1901204544","state":["exists","up"]},{"osd":9,"uuid":"0379f6b0-ec63-487d-92c6-866b04ac385d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":16,"last_clean_end":39,"up_from":40,"up_thru":60,"down_at":36,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6849","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6853","nonce":1516429068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6859","nonce":1517429068},{"type":"v1","addr":"172.21.15.2:6863","nonce":1517429068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6874","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6878","nonce":1516429068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6866","nonce":1516429068},{"type":"v1","addr":"172.21.15.2:6870","nonce":1516429068}]},"public_addr":"172.21.15.2:6853/1516429068","cluster_addr":"172.21.15.2:6863/1517429068","heartbeat_back_addr":"172.21.15.2:6878/1516429068","heartbeat_front_addr":"172.21.15.2:6870/1516429068","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2024-05-08T16:02:16.464404+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.506163+0000","dead_epoch":57},{"osd":1,"down_stamp":"2024-05-08T16:02:16.464404+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.612784+0000","dead_epoch":56},{"osd":2,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.677431+0000","dead_epoch":47},{"osd":3,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.720569+0000","dead_epoch":43},{"osd":4,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.731739+0000","dead_epoch":43},{"osd":5,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.625047+0000","dead_epoch":47},{"osd":6,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.663568+0000","dead_epoch":44},{"osd":7,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.965907+0000","dead_epoch":43},{"osd":8,"down_stamp":"2024-05-08T16:02:03.381281+0000","laggy_probability":0.50999999046325684,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.988031+0000","dead_epoch":43},{"osd":9,"down_stamp":"2024-05-08T16:01:55.330371+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-08T15:59:46.891542+0000","dead_epoch":37}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-08T16:02:28.124 INFO:tasks.ceph:Scrubbing osd.0 2024-05-08T16:02:28.125 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:28.288 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:28.288 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:28.288 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:28.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-08T16:02:28.604 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 0 to deep-scrub 2024-05-08T16:02:28.624 INFO:tasks.ceph:Scrubbing osd.1 2024-05-08T16:02:28.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:28.771 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:28.771 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:28.771 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:28.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-08T16:02:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 1 to deep-scrub 2024-05-08T16:02:29.091 INFO:tasks.ceph:Scrubbing osd.2 2024-05-08T16:02:29.091 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:29.242 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:29.243 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:29.243 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:29.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-08T16:02:29.554 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 2 to deep-scrub 2024-05-08T16:02:29.575 INFO:tasks.ceph:Scrubbing osd.3 2024-05-08T16:02:29.575 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:29.717 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:29.717 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:29.717 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:29.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-08T16:02:30.013 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 3 to deep-scrub 2024-05-08T16:02:30.031 INFO:tasks.ceph:Scrubbing osd.4 2024-05-08T16:02:30.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:30.180 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:30.181 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:30.181 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:30.195 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-08T16:02:30.474 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 4 to deep-scrub 2024-05-08T16:02:30.492 INFO:tasks.ceph:Scrubbing osd.5 2024-05-08T16:02:30.492 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:30.627 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:30.627 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:30.627 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:30.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-08T16:02:30.928 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 5 to deep-scrub 2024-05-08T16:02:30.947 INFO:tasks.ceph:Scrubbing osd.6 2024-05-08T16:02:30.948 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:31.102 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:31.102 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:31.102 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:31.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-08T16:02:31.416 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 6 to deep-scrub 2024-05-08T16:02:31.435 INFO:tasks.ceph:Scrubbing osd.7 2024-05-08T16:02:31.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:31.578 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:31.578 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:31.579 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:31.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-08T16:02:31.878 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 7 to deep-scrub 2024-05-08T16:02:31.899 INFO:tasks.ceph:Scrubbing osd.8 2024-05-08T16:02:31.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:32.052 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:32.052 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:32.052 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:32.069 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-05-08T16:02:32.355 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 8 to deep-scrub 2024-05-08T16:02:32.373 INFO:tasks.ceph:Scrubbing osd.9 2024-05-08T16:02:32.373 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-05-08T16:02:32.525 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-05-08T16:02:32.525 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-08T16:02:32.525 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-05-08T16:02:32.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-05-08T16:02:32.831 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 9 to deep-scrub 2024-05-08T16:02:32.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:33.136 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:33.136 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:33.152 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-05-08T16:02:32.081305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293676,"kb_used_data":6800,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282324,"statfs":{"total":1073741824000,"available":1073441099776,"internally_reserved":0,"allocated":6963200,"data_stored":3375440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.496154+0000","last_change":"2024-05-08T16:02:08.415947+0000","last_active":"2024-05-08T16:02:21.496154+0000","last_peered":"2024-05-08T16:02:21.496154+0000","last_clean":"2024-05-08T16:02:21.496154+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:21.496154+0000","last_undegraded":"2024-05-08T16:02:21.496154+0000","last_fullsized":"2024-05-08T16:02:21.496154+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T16:34:41.115641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522752+0000","last_change":"2024-05-08T16:02:08.415497+0000","last_active":"2024-05-08T16:02:21.522752+0000","last_peered":"2024-05-08T16:02:21.522752+0000","last_clean":"2024-05-08T16:02:21.522752+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:21.522752+0000","last_undegraded":"2024-05-08T16:02:21.522752+0000","last_fullsized":"2024-05-08T16:02:21.522752+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:15:15.456034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.524888+0000","last_change":"2024-05-08T16:02:21.524888+0000","last_active":"2024-05-08T16:02:21.524888+0000","last_peered":"2024-05-08T16:02:21.524888+0000","last_clean":"2024-05-08T16:02:21.524888+0000","last_became_active":"2024-05-08T16:02:21.524452+0000","last_became_peered":"2024-05-08T16:02:21.524452+0000","last_unstale":"2024-05-08T16:02:21.524888+0000","last_undegraded":"2024-05-08T16:02:21.524888+0000","last_fullsized":"2024-05-08T16:02:21.524888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:22:34.099380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.525757+0000","last_change":"2024-05-08T16:02:21.524959+0000","last_active":"2024-05-08T16:02:21.525757+0000","last_peered":"2024-05-08T16:02:21.525757+0000","last_clean":"2024-05-08T16:02:21.525757+0000","last_became_active":"2024-05-08T16:02:21.524534+0000","last_became_peered":"2024-05-08T16:02:21.524534+0000","last_unstale":"2024-05-08T16:02:21.525757+0000","last_undegraded":"2024-05-08T16:02:21.525757+0000","last_fullsized":"2024-05-08T16:02:21.525757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":95,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498540+0000","last_change":"2024-05-08T16:02:21.498540+0000","last_active":"2024-05-08T16:02:21.498540+0000","last_peered":"2024-05-08T16:02:21.498540+0000","last_clean":"2024-05-08T16:02:21.498540+0000","last_became_active":"2024-05-08T16:02:21.498129+0000","last_became_peered":"2024-05-08T16:02:21.498129+0000","last_unstale":"2024-05-08T16:02:21.498540+0000","last_undegraded":"2024-05-08T16:02:21.498540+0000","last_fullsized":"2024-05-08T16:02:21.498540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T22:30:16.891475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":97,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.498390+0000","last_change":"2024-05-08T16:02:21.498390+0000","last_active":"2024-05-08T16:02:21.498390+0000","last_peered":"2024-05-08T16:02:21.498390+0000","last_clean":"2024-05-08T16:02:21.498390+0000","last_became_active":"2024-05-08T16:02:21.498079+0000","last_became_peered":"2024-05-08T16:02:21.498079+0000","last_unstale":"2024-05-08T16:02:21.498390+0000","last_undegraded":"2024-05-08T16:02:21.498390+0000","last_fullsized":"2024-05-08T16:02:21.498390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T18:16:41.125478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.523069+0000","last_change":"2024-05-08T16:02:08.417864+0000","last_active":"2024-05-08T16:02:21.523069+0000","last_peered":"2024-05-08T16:02:21.523069+0000","last_clean":"2024-05-08T16:02:21.523069+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:21.523069+0000","last_undegraded":"2024-05-08T16:02:21.523069+0000","last_fullsized":"2024-05-08T16:02:21.523069+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T21:38:05.966996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":87,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:21.522687+0000","last_change":"2024-05-08T16:02:11.195121+0000","last_active":"2024-05-08T16:02:21.522687+0000","last_peered":"2024-05-08T16:02:21.522687+0000","last_clean":"2024-05-08T16:02:21.522687+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:21.522687+0000","last_undegraded":"2024-05-08T16:02:21.522687+0000","last_fullsized":"2024-05-08T16:02:21.522687+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T15:59:52.141052+0000","last_clean_scrub_stamp":"2024-05-08T15:59:52.141052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T17:50:34.184731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825736,"statfs":{"total":107374182400,"available":107341553664,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462947,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462949,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825740,"statfs":{"total":107374182400,"available":107341557760,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825732,"statfs":{"total":107374182400,"available":107341549568,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462947,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462947,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462947,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397540,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037796,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829892,"statfs":{"total":107374182400,"available":107345809408,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037796,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:33.156 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.156 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.156 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.156 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.156 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.157 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.157 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.157 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-08T15:59:52.141052+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=15, tm_min=59, tm_sec=52, tm_wday=2, tm_yday=129, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=8, tm_hour=16, tm_min=2, tm_sec=27, tm_wday=2, tm_yday=129, tm_isdst=0) 2024-05-08T16:02:33.157 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-08T16:02:53.157 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-08T16:02:53.467 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-05-08T16:02:53.468 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-05-08T16:02:53.487 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-05-08T16:02:52.086396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":293620,"kb_used_data":6800,"kb_used_omap":15,"kb_used_meta":286704,"kb_avail":1048282380,"statfs":{"total":1073741824000,"available":1073441157120,"internally_reserved":0,"allocated":6963200,"data_stored":3375440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15947,"internal_metadata":293585333},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:30.095181+0000","last_change":"2024-05-08T16:02:30.095219+0000","last_active":"2024-05-08T16:02:30.095181+0000","last_peered":"2024-05-08T16:02:30.095181+0000","last_clean":"2024-05-08T16:02:30.095181+0000","last_became_active":"2024-05-08T16:02:08.415212+0000","last_became_peered":"2024-05-08T16:02:08.415212+0000","last_unstale":"2024-05-08T16:02:30.095181+0000","last_undegraded":"2024-05-08T16:02:30.095181+0000","last_fullsized":"2024-05-08T16:02:30.095181+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:30.095088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:30.095088+0000","last_clean_scrub_stamp":"2024-05-08T16:02:30.095088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T00:38:59.081110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":104,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:31.632220+0000","last_change":"2024-05-08T16:02:31.632239+0000","last_active":"2024-05-08T16:02:31.632220+0000","last_peered":"2024-05-08T16:02:31.632220+0000","last_clean":"2024-05-08T16:02:31.632220+0000","last_became_active":"2024-05-08T16:02:08.414590+0000","last_became_peered":"2024-05-08T16:02:08.414590+0000","last_unstale":"2024-05-08T16:02:31.632220+0000","last_undegraded":"2024-05-08T16:02:31.632220+0000","last_fullsized":"2024-05-08T16:02:31.632220+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:31.632162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:31.632162+0000","last_clean_scrub_stamp":"2024-05-08T16:02:31.632162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T18:46:03.820847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":117,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:33.456607+0000","last_change":"2024-05-08T16:02:33.456711+0000","last_active":"2024-05-08T16:02:33.456607+0000","last_peered":"2024-05-08T16:02:33.456607+0000","last_clean":"2024-05-08T16:02:33.456607+0000","last_became_active":"2024-05-08T16:02:21.524452+0000","last_became_peered":"2024-05-08T16:02:21.524452+0000","last_unstale":"2024-05-08T16:02:33.456607+0000","last_undegraded":"2024-05-08T16:02:33.456607+0000","last_fullsized":"2024-05-08T16:02:33.456607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:33.456435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:33.456435+0000","last_clean_scrub_stamp":"2024-05-08T16:02:33.456435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:34:58.327776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:29.798321+0000","last_change":"2024-05-08T16:02:29.798360+0000","last_active":"2024-05-08T16:02:29.798321+0000","last_peered":"2024-05-08T16:02:29.798321+0000","last_clean":"2024-05-08T16:02:29.798321+0000","last_became_active":"2024-05-08T16:02:21.524534+0000","last_became_peered":"2024-05-08T16:02:21.524534+0000","last_unstale":"2024-05-08T16:02:29.798321+0000","last_undegraded":"2024-05-08T16:02:29.798321+0000","last_fullsized":"2024-05-08T16:02:29.798321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:29.798254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:29.798254+0000","last_clean_scrub_stamp":"2024-05-08T16:02:29.798254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T23:18:29.367822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":104,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:31.123244+0000","last_change":"2024-05-08T16:02:31.123262+0000","last_active":"2024-05-08T16:02:31.123244+0000","last_peered":"2024-05-08T16:02:31.123244+0000","last_clean":"2024-05-08T16:02:31.123244+0000","last_became_active":"2024-05-08T16:02:21.498129+0000","last_became_peered":"2024-05-08T16:02:21.498129+0000","last_unstale":"2024-05-08T16:02:31.123244+0000","last_undegraded":"2024-05-08T16:02:31.123244+0000","last_fullsized":"2024-05-08T16:02:31.123244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:31.123188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:31.123188+0000","last_clean_scrub_stamp":"2024-05-08T16:02:31.123188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T01:34:55.994529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":106,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:30.902249+0000","last_change":"2024-05-08T16:02:30.902303+0000","last_active":"2024-05-08T16:02:30.902249+0000","last_peered":"2024-05-08T16:02:30.902249+0000","last_clean":"2024-05-08T16:02:30.902249+0000","last_became_active":"2024-05-08T16:02:21.498079+0000","last_became_peered":"2024-05-08T16:02:21.498079+0000","last_unstale":"2024-05-08T16:02:30.902249+0000","last_undegraded":"2024-05-08T16:02:30.902249+0000","last_fullsized":"2024-05-08T16:02:30.902249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:30.902118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:30.902118+0000","last_clean_scrub_stamp":"2024-05-08T16:02:30.902118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T02:15:17.000239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":89,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:32.043218+0000","last_change":"2024-05-08T16:02:32.043240+0000","last_active":"2024-05-08T16:02:32.043218+0000","last_peered":"2024-05-08T16:02:32.043218+0000","last_clean":"2024-05-08T16:02:32.043218+0000","last_became_active":"2024-05-08T16:02:08.417063+0000","last_became_peered":"2024-05-08T16:02:08.417063+0000","last_unstale":"2024-05-08T16:02:32.043218+0000","last_undegraded":"2024-05-08T16:02:32.043218+0000","last_fullsized":"2024-05-08T16:02:32.043218+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:32.043152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:32.043152+0000","last_clean_scrub_stamp":"2024-05-08T16:02:32.043152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T20:25:14.001480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":96,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-05-08T16:02:31.769972+0000","last_change":"2024-05-08T16:02:31.770010+0000","last_active":"2024-05-08T16:02:31.769972+0000","last_peered":"2024-05-08T16:02:31.769972+0000","last_clean":"2024-05-08T16:02:31.769972+0000","last_became_active":"2024-05-08T16:02:11.194372+0000","last_became_peered":"2024-05-08T16:02:11.194372+0000","last_unstale":"2024-05-08T16:02:31.769972+0000","last_undegraded":"2024-05-08T16:02:31.769972+0000","last_fullsized":"2024-05-08T16:02:31.769972+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-08T16:02:31.769866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-08T16:02:31.769866+0000","last_clean_scrub_stamp":"2024-05-08T16:02:31.769866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-09T18:06:56.264164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":40,"seq":171798691880,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462951,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462953,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":49,"seq":210453397544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":31856,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":104825744,"statfs":{"total":107374182400,"available":107341561856,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":47,"seq":201863462951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":47,"seq":201863462951,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":47,"seq":201863462951,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":49,"seq":210453397544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829896,"statfs":{"total":107374182400,"available":107345813504,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":60,"seq":257698037800,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":60,"seq":257698037800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829900,"statfs":{"total":107374182400,"available":107345817600,"internally_reserved":0,"allocated":696320,"data_stored":337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-08T16:02:53.488 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-08T16:02:53.824 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-08T16:02:53.824 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-08T16:02:53.824 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-08T16:02:53.825 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:53.907 INFO:tasks.ceph.osd.0:Stopped 2024-05-08T16:02:53.907 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-08T16:02:53.907 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:54.004 INFO:tasks.ceph.osd.1:Stopped 2024-05-08T16:02:54.005 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-08T16:02:54.005 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:54.094 INFO:tasks.ceph.osd.2:Stopped 2024-05-08T16:02:54.095 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-08T16:02:54.095 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:54.178 INFO:tasks.ceph.osd.3:Stopped 2024-05-08T16:02:54.178 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-08T16:02:54.178 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:54.261 INFO:tasks.ceph.osd.4:Stopped 2024-05-08T16:02:54.261 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-08T16:02:54.261 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:54.351 INFO:tasks.ceph.osd.5:Stopped 2024-05-08T16:02:54.352 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-08T16:02:54.352 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:02:59.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-05-08T16:03:00.454 INFO:tasks.ceph.osd.6:Stopped 2024-05-08T16:03:00.454 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-08T16:03:00.454 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:03:00.513 INFO:tasks.ceph.osd.7:Stopped 2024-05-08T16:03:00.513 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-05-08T16:03:00.513 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:03:00.611 INFO:tasks.ceph.osd.8:Stopped 2024-05-08T16:03:00.611 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-05-08T16:03:00.611 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:03:00.694 INFO:tasks.ceph.osd.9:Stopped 2024-05-08T16:03:00.694 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-08T16:03:00.695 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-08T16:03:00.695 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:03:00.725 INFO:tasks.ceph.mgr.x:Stopped 2024-05-08T16:03:00.725 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-08T16:03:00.725 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-08T16:03:00.725 INFO:teuthology.orchestra.run:waiting for 300 2024-05-08T16:03:00.784 INFO:tasks.ceph.mon.a:Stopped 2024-05-08T16:03:00.784 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-08T16:03:00.784 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-08T16:03:00.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T16:03:00.840 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-08T16:03:01.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T16:03:01.121 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-08T16:03:01.259 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T16:03:01.259 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-08T16:03:01.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T16:03:01.408 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-08T16:03:01.593 INFO:tasks.ceph:Archiving mon data... 2024-05-08T16:03:01.594 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631/data/mon.a.tgz 2024-05-08T16:03:01.594 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-05-08T16:03:01.600 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.0ld2lkti14 2024-05-08T16:03:01.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0ld2lkti14 2024-05-08T16:03:01.700 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.0ld2lkti14 2024-05-08T16:03:01.770 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.0ld2lkti14 is 219KB 2024-05-08T16:03:01.827 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.0ld2lkti14 2024-05-08T16:03:01.832 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-08T16:03:01.832 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-08T16:03:01.938 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-08T16:03:01.938 INFO:tasks.ceph:Archiving crash dumps... 2024-05-08T16:03:01.939 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631/remote/smithi002/crash 2024-05-08T16:03:01.940 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-08T16:03:01.993 INFO:tasks.ceph:Compressing logs... 2024-05-08T16:03:01.994 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-08T16:03:02.066 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36576.log 2024-05-08T16:03:02.067 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36922.log 2024-05-08T16:03:02.067 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36429.log 2024-05-08T16:03:02.067 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36576.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36453.log 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36922.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36304.log 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36429.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34976.log 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36453.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35637.log 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36304.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34976.log.gz 2024-05-08T16:03:02.070 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-05-08T16:03:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36848.log 2024-05-08T16:03:02.071 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35637.log.gz 2024-05-08T16:03:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-05-08T16:03:02.071 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36948.log 2024-05-08T16:03:02.086 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36848.log: /var/log/ceph/ceph-client.admin.36948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36354.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36848.log.gz 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36948.log.gz 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36700.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35382.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36354.log.gz 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28254.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36700.log.gz 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37382.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35382.log.gz 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-05-08T16:03:02.087 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28254.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37168.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37382.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36626.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35127.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37168.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35139.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36626.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35141.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35127.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35139.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35139.log.gz 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37980.log 2024-05-08T16:03:02.088 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35141.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37434.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37980.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37980.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37434.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37434.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28350.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28350.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35711.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37460.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.35711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35711.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35354.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37460.log.gz 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28095.log 2024-05-08T16:03:02.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37694.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35354.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28095.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36651.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37694.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37694.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35886.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37538.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36651.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36279.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35886.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36725.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37538.log.gz 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28374.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36279.log: /var/log/ceph/ceph-client.admin.36725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35388.log 2024-05-08T16:03:02.090 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36279.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36725.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28374.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37590.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28374.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35388.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35388.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36798.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37590.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35699.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36526.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36798.log.gz 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35699.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36675.log 2024-05-08T16:03:02.091 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36526.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36675.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37330.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38030.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37330.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35118.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38030.log: /var/log/ceph/ceph-client.admin.35118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38030.log.gz 2024-05-08T16:03:02.097 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37217.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37824.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37798.log: /var/log/ceph/ceph-client.admin.37217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37217.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28422.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37928.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37824.log: /var/log/ceph/ceph-client.admin.28422.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37824.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28422.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37720.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37928.log: /var/log/ceph/ceph-client.admin.37720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37928.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37720.log.gz 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37252.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35125.log 2024-05-08T16:03:02.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37045.log: /var/log/ceph/ceph-client.admin.37252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-05-08T16:03:02.099 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37252.log.gz 2024-05-08T16:03:02.099 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-05-08T16:03:02.100 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-08T16:03:02.100 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35125.log: /var/log/ceph/ceph-client.admin.36750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35125.log.gz 2024-05-08T16:03:02.100 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-05-08T16:03:02.100 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-08T16:03:02.101 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-05-08T16:03:02.113 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36230.log 2024-05-08T16:03:02.113 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-05-08T16:03:02.114 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28747.log 2024-05-08T16:03:02.114 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36230.log.gz 2024-05-08T16:03:02.121 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38006.log 2024-05-08T16:03:02.121 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28747.log.gz 2024-05-08T16:03:02.137 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37408.log 2024-05-08T16:03:02.137 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38006.log.gz 2024-05-08T16:03:02.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35364.log 2024-05-08T16:03:02.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37408.log.gz 2024-05-08T16:03:02.153 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-05-08T16:03:02.154 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35364.log.gz 2024-05-08T16:03:02.154 INFO:teuthology.orchestra.run.smithi002.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-08T16:03:02.155 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-08T16:03:02.156 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-05-08T16:03:02.156 INFO:teuthology.orchestra.run.smithi002.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-05-08T16:03:02.156 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37069.log 2024-05-08T16:03:02.161 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.1.log: 91.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-08T16:03:02.162 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37512.log 2024-05-08T16:03:02.163 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-05-08T16:03:02.164 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37069.log.gz 2024-05-08T16:03:02.164 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37512.log.gz 2024-05-08T16:03:02.164 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35938.log 2024-05-08T16:03:02.164 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35655.log 2024-05-08T16:03:02.164 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36477.log 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35938.log.gz 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35655.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35445.log 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36501.log 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0%/var/log/ceph/ceph-client.admin.35445.log: -- replaced with /var/log/ceph/ceph-client.admin.36477.log.gz 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35445.log.gz 2024-05-08T16:03:02.166 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35912.log 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36501.log.gz 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37564.log 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35912.log.gz 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35130.log 2024-05-08T16:03:02.169 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-08T16:03:02.170 INFO:teuthology.orchestra.run.smithi002.stderr: 92.0%/var/log/ceph/ceph-client.admin.37564.log: -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-08T16:03:02.170 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37564.log.gz 2024-05-08T16:03:02.170 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35130.log.gz 2024-05-08T16:03:02.170 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36182.log 2024-05-08T16:03:02.170 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28278.log 2024-05-08T16:03:02.171 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37902.log 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36182.log.gz 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28278.log.gz 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37902.log.gz 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-05-08T16:03:02.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36873.log 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35394.log 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36873.log.gz 92.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-08T16:03:02.179 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36378.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0%/var/log/ceph/ceph-client.admin.35378.log: -- replaced with /var/log/ceph/ceph-client.admin.35394.log.gz 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28515.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35638.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36378.log.gz 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37143.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28515.log.gz 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35131.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35638.log: 0.0%/var/log/ceph/ceph-client.admin.37143.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.35638.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.36997.log 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37143.log.gz 2024-05-08T16:03:02.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37486.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35131.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36551.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36997.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37486.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37486.log.gz -5 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35636.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37093.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36551.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35636.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28302.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37093.log: 0.0% 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37954.log: -- replaced with /var/log/ceph/ceph-client.admin.37093.log.gz 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37304.log 2024-05-08T16:03:02.181 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28302.log: /var/log/ceph/ceph-client.admin.37304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28302.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20196.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37304.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37746.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35133.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.tmp-client.admin.20196.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20196.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37746.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.35133.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35363.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35133.log.gz 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36898.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-08T16:03:02.191 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37193.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35363.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36206.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36082.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37193.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36206.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr: 89.5% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36206.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37668.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36082.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36823.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35142.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37668.log.gz 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28453.log 2024-05-08T16:03:02.192 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36823.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36106.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35142.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28453.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37772.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36106.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37850.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37772.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35136.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37850.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37876.log 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35390.log: /var/log/ceph/ceph-client.admin.35136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-05-08T16:03:02.193 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35136.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37356.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36404.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37876.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.37356.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36254.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37356.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36404.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36254.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36054.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36254.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37118.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz/var/log/ceph/ceph-client.admin.36054.log: gzip 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36054.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37118.log.gz 2024-05-08T16:03:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36329.log 2024-05-08T16:03:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34951.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.36329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35626.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36329.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34951.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35656.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35646.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35626.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35656.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28326.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28398.log 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35646.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28326.log.gz 2024-05-08T16:03:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35002.log 2024-05-08T16:03:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28230.log 2024-05-08T16:03:02.199 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28398.log.gz 2024-05-08T16:03:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.35002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35002.log.gz 2024-05-08T16:03:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.28230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28230.log.gz 2024-05-08T16:03:02.201 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-08T16:03:02.209 INFO:teuthology.orchestra.run.smithi002.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-08T16:03:02.238 INFO:teuthology.orchestra.run.smithi002.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-08T16:03:02.250 INFO:teuthology.orchestra.run.smithi002.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-08T16:03:02.254 INFO:teuthology.orchestra.run.smithi002.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-05-08T16:03:02.265 INFO:teuthology.orchestra.run.smithi002.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-08T16:03:02.266 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-05-08T16:03:02.266 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m0.227s 2024-05-08T16:03:02.266 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m1.102s 2024-05-08T16:03:02.267 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.101s 2024-05-08T16:03:02.267 INFO:tasks.ceph:Archiving logs... 2024-05-08T16:03:02.267 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631/remote/smithi002/log 2024-05-08T16:03:02.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-08T16:03:02.532 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-08T16:03:02.548 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-08T16:03:02.548 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T16:03:02.589 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-08T16:03:02.598 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-08T16:03:02.598 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T16:03:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2024-05-08T16:03:02.622 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2024-05-08T16:03:02.623 INFO:teuthology.orchestra.run.smithi002.stdout:*hv01.front.sepi 67.205.162.81 3 u 29 64 77 0.068 -1.555 2.565 2024-05-08T16:03:02.623 INFO:teuthology.orchestra.run.smithi002.stdout:+hv02.front.sepi 63.231.80.2 3 u 29 64 77 0.068 -2.433 2.042 2024-05-08T16:03:02.623 INFO:teuthology.orchestra.run.smithi002.stdout:+hv03.front.sepi 74.6.168.72 3 u 32 64 77 0.104 -4.483 2.680 2024-05-08T16:03:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T16:03:02.623 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-08T16:03:02.633 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-08T16:03:02.634 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-08T16:03:02.642 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-08T16:03:02.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-08T16:03:02.660 INFO:teuthology.task.internal:Duration was 800.340921 seconds 2024-05-08T16:03:02.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-08T16:03:02.669 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-08T16:03:02.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T16:03:02.707 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-08T16:03:02.707 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-05-08T16:03:02.707 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T16:03:02.758 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-08T16:03:02.759 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T16:03:02.826 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-08T16:03:02.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T16:03:02.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-08T16:03:02.941 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-08T16:03:02.941 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T16:03:02.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-08T16:03:02.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:03.007 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-05-08T16:03:03.022 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T16:03:03.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-08T16:03:03.080 INFO:teuthology.task.internal:Transferring archived files... 2024-05-08T16:03:03.081 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697631/remote/smithi002 2024-05-08T16:03:03.081 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T16:03:03.133 INFO:teuthology.task.internal:Removing archive directory... 2024-05-08T16:03:03.133 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T16:03:03.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-08T16:03:03.190 INFO:teuthology.task.internal:Not uploading archives. 2024-05-08T16:03:03.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-08T16:03:03.200 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-08T16:03:03.200 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T16:03:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: 658618 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 16:03 /home/ubuntu/cephtest 2024-05-08T16:03:03.228 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-08T16:03:03.238 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-08T16:03:03.263 INFO:teuthology.run:Summary data: description: rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} duration: 800.3409209251404 flavor: default owner: scheduled_lflores@teuthology success: true 2024-05-08T16:03:03.263 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T16:03:03.345 INFO:teuthology.run:pass