2023-09-23T02:27:19.112 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-23T02:27:19.112 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rraja@teuthology --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --name rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi --archive /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437 -- /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437/orig.config.yaml 2023-09-23T02:27:19.145 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-23T02:27:19.217 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437 branch: wip-62891 description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7401437' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-62891 ceph: conf: client: rbd default data pool: datapool rbd default features: 61 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 enable experimental unrecoverable data corrupting features: '*' mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug randomize hobject sort order: false osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-62891 sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e owner: scheduled_rraja@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3804 sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-62891 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTHhvD+lHCTnuCMxkFMRGgP2c+8SiXjzcY+NkKI2aVnWsEVb0yz3qCxaMsEyq5Or+5R29zNh9dNSgt0PSx0Ozc= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2 - sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile - sudo ceph osd pool set datapool allow_ec_overwrites true - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-23_02:20:47 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-23T02:27:19.217 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa; will attempt to use it 2023-09-23T02:27:19.218 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks 2023-09-23T02:27:19.218 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-23T02:27:19.220 INFO:teuthology.task.internal:Checking packages... 2023-09-23T02:27:19.240 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e' 2023-09-23T02:27:19.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-23T02:27:19.241 INFO:teuthology.packaging:ref: None 2023-09-23T02:27:19.241 INFO:teuthology.packaging:tag: None 2023-09-23T02:27:19.241 INFO:teuthology.packaging:branch: wip-62891 2023-09-23T02:27:19.241 INFO:teuthology.packaging:sha1: ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:27:19.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-62891 2023-09-23T02:27:19.460 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6334-gee696c23-1jammy 2023-09-23T02:27:19.461 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-23T02:27:19.467 INFO:teuthology.task.internal:no buildpackages task found 2023-09-23T02:27:19.467 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-23T02:27:19.478 INFO:teuthology.task.internal:Saving configuration 2023-09-23T02:27:19.490 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-23T02:27:19.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-23T02:27:19.518 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-23 02:21:50.815094', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINIYvPezts+a5OF6wK0ZTebF5TctDSeJ6HlBt/KspCB4'} 2023-09-23T02:27:19.518 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-23T02:27:19.524 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-23T02:27:19.524 INFO:teuthology.run_tasks:Running task console_log... 2023-09-23T02:27:19.565 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fba5ff4faf0>, signals=[15]) 2023-09-23T02:27:19.566 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-23T02:27:19.572 INFO:teuthology.task.internal:Opening connections... 2023-09-23T02:27:19.572 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:27:19.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T02:27:19.649 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-23T02:27:19.656 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-09-23T02:27:19.760 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-09-23T02:27:19.760 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-09-23T02:27:19.807 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-09-23T02:27:19.807 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Ubuntu" 2023-09-23T02:27:19.807 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="22.04" 2023-09-23T02:27:19.807 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_CODENAME=jammy 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:ID=ubuntu 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE=debian 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-23T02:27:19.808 INFO:teuthology.orchestra.run.smithi002.stdout:UBUNTU_CODENAME=jammy 2023-09-23T02:27:19.809 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-09-23T02:27:19.829 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-23T02:27:19.838 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-23T02:27:19.891 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-23T02:27:19.891 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-09-23T02:27:19.895 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-23T02:27:19.901 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-23T02:27:19.901 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-09-23T02:27:19.945 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-23T02:27:19.974 INFO:teuthology.run_tasks:Running task kernel... 2023-09-23T02:27:19.988 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-23T02:27:19.988 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-23T02:27:19.988 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-23T02:27:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T02:27:19.993 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-09-23T02:27:20.040 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-56-generic 2023-09-23T02:27:20.040 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.15.0-56-generic 2023-09-23T02:27:20.040 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2023-09-23T02:27:20.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-09-23T02:27:20.351 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-09-23T02:27:20.433 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-23T02:27:20.515 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-09-23T02:27:20.570 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [328 kB] 2023-09-23T02:27:20.640 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [802 kB] 2023-09-23T02:27:20.675 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [169 kB] 2023-09-23T02:27:20.680 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.3 kB] 2023-09-23T02:27:20.680 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [30.0 kB] 2023-09-23T02:27:20.681 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [882 kB] 2023-09-23T02:27:20.694 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [142 kB] 2023-09-23T02:27:20.697 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-09-23T02:27:20.697 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [785 kB] 2023-09-23T02:27:20.707 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [559 kB] 2023-09-23T02:27:20.712 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [143 kB] 2023-09-23T02:27:20.713 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-09-23T02:27:20.715 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-09-23T02:27:20.730 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-09-23T02:27:20.730 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-09-23T02:27:20.730 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-09-23T02:27:20.867 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-09-23T02:27:21.005 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,012 kB] 2023-09-23T02:27:21.268 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [493 kB] 2023-09-23T02:27:21.308 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [227 kB] 2023-09-23T02:27:21.323 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [15.6 kB] 2023-09-23T02:27:21.324 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [30.3 kB] 2023-09-23T02:27:21.327 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [898 kB] 2023-09-23T02:27:21.377 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [145 kB] 2023-09-23T02:27:21.383 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-09-23T02:27:21.547 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [655 kB] 2023-09-23T02:27:22.119 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [984 kB] 2023-09-23T02:27:22.228 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [215 kB] 2023-09-23T02:27:22.248 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.8 kB] 2023-09-23T02:27:22.249 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-09-23T02:27:22.253 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-09-23T02:27:22.253 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-09-23T02:27:22.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [476 B] 2023-09-23T02:27:22.255 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-09-23T02:27:22.257 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-09-23T02:27:22.260 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-09-23T02:27:22.425 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-09-23T02:27:22.508 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-09-23T02:27:22.513 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-09-23T02:27:22.632 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-09-23T02:27:22.684 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-09-23T02:27:27.067 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 9,144 kB in 2s (3,710 kB/s) 2023-09-23T02:27:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:27:28.407 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-23T02:27:28.517 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:27:28.705 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-09-23T02:27:28.706 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-09-23T02:27:28.868 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout: python2.7-minimal 2023-09-23T02:27:28.869 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-23T02:27:28.870 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2023-09-23T02:27:28.870 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-23T02:27:28.870 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-generic linux-image-5.15.0-84-generic 2023-09-23T02:27:28.870 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-5.15.0-84-generic linux-modules-extra-5.15.0-84-generic 2023-09-23T02:27:28.872 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2023-09-23T02:27:28.872 INFO:teuthology.orchestra.run.smithi002.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-09-23T02:27:28.872 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2023-09-23T02:27:28.872 INFO:teuthology.orchestra.run.smithi002.stdout: thermald 2023-09-23T02:27:28.912 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2023-09-23T02:27:28.912 INFO:teuthology.orchestra.run.smithi002.stdout: linux-headers-5.15.0-84 linux-headers-5.15.0-84-generic 2023-09-23T02:27:28.912 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.15.0-84-generic linux-modules-5.15.0-84-generic 2023-09-23T02:27:28.912 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-extra-5.15.0-84-generic 2023-09-23T02:27:28.914 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages will be upgraded: 2023-09-23T02:27:28.914 INFO:teuthology.orchestra.run.smithi002.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-23T02:27:28.970 INFO:teuthology.orchestra.run.smithi002.stdout:3 upgraded, 5 newly installed, 0 to remove and 265 not upgraded. 2023-09-23T02:27:28.970 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 113 MB of archives. 2023-09-23T02:27:28.970 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 582 MB of additional disk space will be used. 2023-09-23T02:27:28.970 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-84-generic amd64 5.15.0-84.93 [22.5 MB] 2023-09-23T02:27:29.425 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-84-generic amd64 5.15.0-84.93 [11.5 MB] 2023-09-23T02:27:29.525 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-84-generic amd64 5.15.0-84.93 [63.9 MB] 2023-09-23T02:27:30.390 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.84.81 [1,698 B] 2023-09-23T02:27:30.390 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.84.81 [2,476 B] 2023-09-23T02:27:30.390 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84 all 5.15.0-84.93 [12.3 MB] 2023-09-23T02:27:30.665 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-84-generic amd64 5.15.0-84.93 [2,853 kB] 2023-09-23T02:27:30.702 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.84.81 [2,330 B] 2023-09-23T02:27:31.111 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 113 MB in 2s (63.5 MB/s) 2023-09-23T02:27:31.261 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-5.15.0-84-generic. 2023-09-23T02:27:33.219 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-09-23T02:27:33.223 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../0-linux-modules-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-23T02:27:33.300 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:35.267 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-image-5.15.0-84-generic. 2023-09-23T02:27:35.288 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../1-linux-image-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-23T02:27:35.492 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:35.954 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-84-generic. 2023-09-23T02:27:35.974 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-23T02:27:36.011 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:42.222 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../3-linux-generic_5.15.0.84.81_amd64.deb ... 2023-09-23T02:27:42.348 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-23T02:27:42.683 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.84.81_amd64.deb ... 2023-09-23T02:27:42.818 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-image-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-23T02:27:43.121 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-84. 2023-09-23T02:27:43.132 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../5-linux-headers-5.15.0-84_5.15.0-84.93_all.deb ... 2023-09-23T02:27:43.171 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-23T02:27:48.302 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package linux-headers-5.15.0-84-generic. 2023-09-23T02:27:48.340 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../6-linux-headers-5.15.0-84-generic_5.15.0-84.93_amd64.deb ... 2023-09-23T02:27:48.376 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:51.009 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.84.81_amd64.deb ... 2023-09-23T02:27:51.131 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking linux-headers-generic (5.15.0.84.81) over (5.15.0.56.54) ... 2023-09-23T02:27:51.465 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-84 (5.15.0-84.93) ... 2023-09-23T02:27:51.586 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:51.712 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-headers-generic (5.15.0.84.81) ... 2023-09-23T02:27:51.840 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:53.202 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-09-23T02:27:53.202 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-09-23T02:27:53.202 INFO:teuthology.orchestra.run.smithi002.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-84-generic 2023-09-23T02:27:53.202 INFO:teuthology.orchestra.run.smithi002.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-84-generic 2023-09-23T02:27:53.359 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-extra-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:54.802 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-modules-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:56.270 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-image-generic (5.15.0.84.81) ... 2023-09-23T02:27:56.396 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up linux-generic (5.15.0.84.81) ... 2023-09-23T02:27:56.505 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for linux-image-5.15.0-84-generic (5.15.0-84.93) ... 2023-09-23T02:27:56.565 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-23T02:27:56.565 INFO:teuthology.orchestra.run.smithi002.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic 2023-09-23T02:28:11.203 INFO:teuthology.orchestra.run.smithi002.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-23T02:28:11.204 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub' 2023-09-23T02:28:11.220 INFO:teuthology.orchestra.run.smithi002.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-23T02:28:11.250 INFO:teuthology.orchestra.run.smithi002.stdout:Generating grub configuration file ... 2023-09-23T02:28:11.593 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-23T02:28:11.609 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-23T02:28:11.909 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-23T02:28:11.915 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-23T02:28:11.986 INFO:teuthology.orchestra.run.smithi002.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-23T02:28:11.991 INFO:teuthology.orchestra.run.smithi002.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-23T02:28:12.162 INFO:teuthology.orchestra.run.smithi002.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-23T02:28:12.162 INFO:teuthology.orchestra.run.smithi002.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-09-23T02:28:12.162 INFO:teuthology.orchestra.run.smithi002.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-23T02:28:12.197 INFO:teuthology.orchestra.run.smithi002.stdout:done 2023-09-23T02:28:12.666 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2023-09-23T02:28:12.691 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2023-09-23T02:28:12.691 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 20 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2023-09-23T02:28:12.692 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.84.81 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2023-09-23T02:28:12.693 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2023-09-23T02:28:12.694 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-84-generic 2023-09-23T02:28:12.694 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-23T02:28:12.694 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-23T02:28:12.694 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-23T02:28:12.695 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2023-09-23T02:28:12.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get clean 2023-09-23T02:28:12.843 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-09-23T02:28:12.961 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-23T02:28:12.969 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-23T02:28:12.986 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-23T02:28:13.080 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-23T02:28:14.250 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:28:14.270 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-23T02:28:14.333 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:28:14.509 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-09-23T02:28:14.510 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-09-23T02:28:14.673 INFO:teuthology.orchestra.run.smithi002.stdout:linux-image-generic is already the newest version (5.15.0.84.81). 2023-09-23T02:28:14.673 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-09-23T02:28:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-23T02:28:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-23T02:28:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-23T02:28:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-23T02:28:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-23T02:28:14.674 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-23T02:28:14.714 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 0 newly installed, 0 to remove and 265 not upgraded. 2023-09-23T02:28:14.716 DEBUG:teuthology.orchestra.run.smithi002:> dpkg -s linux-image-generic 2023-09-23T02:28:14.733 INFO:teuthology.orchestra.run.smithi002.stdout:Package: linux-image-generic 2023-09-23T02:28:14.733 INFO:teuthology.orchestra.run.smithi002.stdout:Status: install ok installed 2023-09-23T02:28:14.733 INFO:teuthology.orchestra.run.smithi002.stdout:Priority: optional 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Section: kernel 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Installed-Size: 20 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Maintainer: Ubuntu Kernel Team 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: amd64 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Source: linux-meta 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Version: 5.15.0.84.81 2023-09-23T02:28:14.734 INFO:teuthology.orchestra.run.smithi002.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-84), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-09-23T02:28:14.735 INFO:teuthology.orchestra.run.smithi002.stdout:Depends: linux-image-5.15.0-84-generic, linux-modules-extra-5.15.0-84-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-23T02:28:14.735 INFO:teuthology.orchestra.run.smithi002.stdout:Recommends: thermald 2023-09-23T02:28:14.735 INFO:teuthology.orchestra.run.smithi002.stdout:Description: Generic Linux kernel image 2023-09-23T02:28:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: This package will always depend on the latest generic kernel image 2023-09-23T02:28:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: available. 2023-09-23T02:28:14.735 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-09-23T02:28:14.779 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.fcOVUOJok1 2023-09-23T02:28:14.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub/grub.cfg /tmp/tmp.fcOVUOJok1 2023-09-23T02:28:14.835 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.fcOVUOJok1 2023-09-23T02:28:14.967 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.fcOVUOJok1 is 10KB 2023-09-23T02:28:14.982 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.fcOVUOJok1 2023-09-23T02:28:14.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-23T02:28:15.043 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:28:15.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-23T02:28:15.044 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-23T02:28:15.110 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-84-generic 2023-09-23T02:28:15.111 DEBUG:teuthology.orchestra.run.smithi002:> sudo update-grub 2023-09-23T02:28:16.135 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub' 2023-09-23T02:28:16.136 INFO:teuthology.orchestra.run.smithi002.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-23T02:28:16.138 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2023-09-23T02:28:16.318 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-84-generic 2023-09-23T02:28:16.324 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-84-generic 2023-09-23T02:28:16.586 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-09-23T02:28:16.593 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-09-23T02:28:16.637 INFO:teuthology.orchestra.run.smithi002.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-09-23T02:28:16.643 INFO:teuthology.orchestra.run.smithi002.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-09-23T02:28:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-09-23T02:28:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-09-23T02:28:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-09-23T02:28:16.781 INFO:teuthology.orchestra.run.smithi002.stderr:done 2023-09-23T02:28:16.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2023-09-23T02:28:46.790 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-23T02:28:46.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-09-23T02:28:46.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T02:29:46.798 DEBUG:teuthology.orchestra.remote:timed out 2023-09-23T02:29:55.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-09-23T02:29:55.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T02:29:56.410 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-09-23T02:29:57.197 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-09-23T02:29:57.198 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-84-generic"... 2023-09-23T02:29:57.198 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-09-23T02:29:57.242 INFO:teuthology.orchestra.run.smithi002.stdout:5.15.0-84-generic 2023-09-23T02:29:57.243 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-84-generic vs 5.15.0-84-generic 2023-09-23T02:29:57.243 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-23T02:29:57.243 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-23T02:29:58.243 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-23T02:29:58.244 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-23T02:29:58.316 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2023-09-23T02:29:58.341 DEBUG:teuthology.parallel:result is None 2023-09-23T02:29:58.341 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-23T02:29:58.349 INFO:teuthology.task.internal:Creating test directory... 2023-09-23T02:29:58.349 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-23T02:29:58.354 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-23T02:29:58.367 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-23T02:29:58.374 INFO:teuthology.task.internal:Creating archive directory... 2023-09-23T02:29:58.375 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-23T02:29:58.425 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-23T02:29:58.432 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-23T02:29:58.432 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-23T02:29:58.493 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-23T02:29:58.501 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-23T02:29:58.502 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-23T02:29:58.509 INFO:teuthology.task.internal:Configuring sudo... 2023-09-23T02:29:58.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-23T02:29:58.559 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-23T02:29:58.568 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-23T02:29:58.568 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-23T02:29:58.608 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-23T02:29:58.655 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-23T02:29:58.704 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:29:58.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-23T02:29:58.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-09-23T02:29:58.906 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-23T02:29:58.913 INFO:teuthology.task.internal:Starting timer... 2023-09-23T02:29:58.913 INFO:teuthology.run_tasks:Running task pcp... 2023-09-23T02:29:58.925 INFO:teuthology.run_tasks:Running task selinux... 2023-09-23T02:29:58.933 DEBUG:teuthology.task.selinux:Excluding smithi002: OS 'ubuntu' does not support SELinux 2023-09-23T02:29:58.933 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-23T02:29:58.933 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-23T02:29:58.934 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-23T02:29:58.934 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-23T02:29:58.942 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-23T02:29:58.942 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-23T02:29:58.956 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-23T02:29:58.957 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-23T02:37:30.850 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com')] 2023-09-23T02:37:30.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-09-23T02:37:30.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-23T02:37:30.931 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-09-23T02:37:31.047 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-09-23T02:37:31.047 INFO:teuthology.run_tasks:Running task clock... 2023-09-23T02:37:31.057 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-23T02:37:31.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-23T02:37:31.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Command line: ntpd -gq 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: ---------------------------------------------------- 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: ntp-4 is maintained by Network Time Foundation, 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: corporation. Support and training for ntp-4 are 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: available at https://www.nwtime.org/support 2023-09-23T02:37:31.117 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: ---------------------------------------------------- 2023-09-23T02:37:31.118 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: proto: precision = 0.064 usec (-24) 2023-09-23T02:37:31.118 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: basedate set to 2022-02-04 2023-09-23T02:37:31.118 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: gps base set to 2022-02-06 (week 2196) 2023-09-23T02:37:31.118 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-23T02:37:31.119 INFO:teuthology.orchestra.run.smithi002.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-23T02:37:31.119 INFO:teuthology.orchestra.run.smithi002.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: restrict ::: KOD does nothing without LIMITED. 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen and drop on 0 v6wildcard [::]:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen normally on 2 lo 127.0.0.1:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen normally on 3 ens1f0 172.21.15.2:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen normally on 4 lo [::1]:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1920%3]:123 2023-09-23T02:37:31.120 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:31 ntpd[27719]: Listening on routing socket on fd #22 for interface updates 2023-09-23T02:37:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:32.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:32 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:34.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:34.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:34.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:34.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:34.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:34.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:34.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:34.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:34.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:34.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:34.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:34.122 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:34.122 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:34.122 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:34 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:35.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:35.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:35.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:35.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:35.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:35.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:35 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:36 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:37.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:37 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:38.118 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: can't open /var/log/ntpstats/rawstats.20230923: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-23T02:37:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: can't open /var/log/ntpstats/peerstats.20230923: Permission denied 2023-09-23T02:37:38.665 INFO:teuthology.orchestra.run.smithi002.stdout:23 Sep 02:37:38 ntpd[27719]: ntpd: time slew -0.000433 s 2023-09-23T02:37:38.665 INFO:teuthology.orchestra.run.smithi002.stdout:ntpd: time slew -0.000433s 2023-09-23T02:37:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-23T02:37:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:23 Sep 02:37:38 ntpd[27719]: can't open /var/log/ntpstats/loopstats.20230923: Permission denied 2023-09-23T02:37:38.666 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2023-09-23T02:37:38.666 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2023-09-23T02:37:38.666 INFO:teuthology.orchestra.run.smithi002.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-23T02:37:38.667 INFO:teuthology.orchestra.run.smithi002.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-23T02:37:38.667 INFO:teuthology.orchestra.run.smithi002.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-23T02:37:38.667 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-09-23T02:37:38.667 INFO:teuthology.run_tasks:Running task install... 2023-09-23T02:37:38.676 DEBUG:teuthology.task.install:project ceph 2023-09-23T02:37:38.676 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}} 2023-09-23T02:37:38.676 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'} 2023-09-23T02:37:38.676 INFO:teuthology.task.install:Using flavor: default 2023-09-23T02:37:38.682 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-23T02:37:38.682 INFO:teuthology.task.install:extra packages: [] 2023-09-23T02:37:38.682 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-key list | grep Ceph 2023-09-23T02:37:38.776 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-09-23T02:37:38.807 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-23T02:37:38.807 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph.com (release key) 2023-09-23T02:37:38.808 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-23T02:37:38.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:37:38.953 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default/ 2023-09-23T02:37:38.954 INFO:teuthology.task.install.deb:Package version is 18.0.0-6334-gee696c23-1jammy 2023-09-23T02:37:39.150 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:37:39.150 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-23T02:37:39.164 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-09-23T02:37:39.337 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-23T02:37:39.346 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-23T02:37:39.354 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-23T02:37:39.363 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-23T02:37:39.462 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy InRelease 2023-09-23T02:37:39.503 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-09-23T02:37:39.539 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy Release.gpg 2023-09-23T02:37:39.587 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-09-23T02:37:39.631 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,419 B] 2023-09-23T02:37:39.721 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 28.6 kB in 0s (76.8 kB/s) 2023-09-23T02:37:40.570 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:37:40.586 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6334-gee696c23-1jammy cephadm=18.0.0-6334-gee696c23-1jammy ceph-mds=18.0.0-6334-gee696c23-1jammy ceph-mgr=18.0.0-6334-gee696c23-1jammy ceph-common=18.0.0-6334-gee696c23-1jammy ceph-fuse=18.0.0-6334-gee696c23-1jammy ceph-test=18.0.0-6334-gee696c23-1jammy ceph-volume=18.0.0-6334-gee696c23-1jammy radosgw=18.0.0-6334-gee696c23-1jammy python3-rados=18.0.0-6334-gee696c23-1jammy python3-rgw=18.0.0-6334-gee696c23-1jammy python3-cephfs=18.0.0-6334-gee696c23-1jammy python3-rbd=18.0.0-6334-gee696c23-1jammy libcephfs2=18.0.0-6334-gee696c23-1jammy libcephfs-dev=18.0.0-6334-gee696c23-1jammy librados2=18.0.0-6334-gee696c23-1jammy librbd1=18.0.0-6334-gee696c23-1jammy rbd-fuse=18.0.0-6334-gee696c23-1jammy 2023-09-23T02:37:40.640 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:37:40.806 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-09-23T02:37:40.807 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-09-23T02:37:40.959 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-23T02:37:40.960 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-23T02:37:40.961 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-09-23T02:37:40.962 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-09-23T02:37:40.963 INFO:teuthology.orchestra.run.smithi002.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-23T02:37:40.964 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-09-23T02:37:40.965 INFO:teuthology.orchestra.run.smithi002.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-09-23T02:37:40.966 INFO:teuthology.orchestra.run.smithi002.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-09-23T02:37:40.966 INFO:teuthology.orchestra.run.smithi002.stdout: python3-watchdog 2023-09-23T02:37:40.966 INFO:teuthology.orchestra.run.smithi002.stdout:Recommended packages: 2023-09-23T02:37:40.966 INFO:teuthology.orchestra.run.smithi002.stdout: btrfs-tools 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-09-23T02:37:41.061 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-09-23T02:37:41.062 INFO:teuthology.orchestra.run.smithi002.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-09-23T02:37:41.062 INFO:teuthology.orchestra.run.smithi002.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-09-23T02:37:41.062 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-09-23T02:37:41.063 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-09-23T02:37:41.064 INFO:teuthology.orchestra.run.smithi002.stdout: socat xmlstarlet 2023-09-23T02:37:41.119 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 109 newly installed, 0 to remove and 253 not upgraded. 2023-09-23T02:37:41.119 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 183 MB of archives. 2023-09-23T02:37:41.119 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 823 MB of additional disk space will be used. 2023-09-23T02:37:41.119 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-09-23T02:37:41.176 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-09-23T02:37:41.212 INFO:teuthology.orchestra.run.smithi002.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-09-23T02:37:41.215 INFO:teuthology.orchestra.run.smithi002.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-09-23T02:37:41.218 INFO:teuthology.orchestra.run.smithi002.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6334-gee696c23-1jammy [3,438 kB] 2023-09-23T02:37:41.230 INFO:teuthology.orchestra.run.smithi002.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-09-23T02:37:41.278 INFO:teuthology.orchestra.run.smithi002.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-09-23T02:37:41.283 INFO:teuthology.orchestra.run.smithi002.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-09-23T02:37:41.296 INFO:teuthology.orchestra.run.smithi002.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-09-23T02:37:41.300 INFO:teuthology.orchestra.run.smithi002.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-09-23T02:37:41.300 INFO:teuthology.orchestra.run.smithi002.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-09-23T02:37:41.301 INFO:teuthology.orchestra.run.smithi002.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-09-23T02:37:41.309 INFO:teuthology.orchestra.run.smithi002.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-09-23T02:37:41.310 INFO:teuthology.orchestra.run.smithi002.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-09-23T02:37:41.311 INFO:teuthology.orchestra.run.smithi002.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-09-23T02:37:41.312 INFO:teuthology.orchestra.run.smithi002.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-09-23T02:37:41.316 INFO:teuthology.orchestra.run.smithi002.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-09-23T02:37:41.317 INFO:teuthology.orchestra.run.smithi002.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-09-23T02:37:41.333 INFO:teuthology.orchestra.run.smithi002.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-09-23T02:37:41.334 INFO:teuthology.orchestra.run.smithi002.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-09-23T02:37:41.334 INFO:teuthology.orchestra.run.smithi002.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-09-23T02:37:41.335 INFO:teuthology.orchestra.run.smithi002.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-09-23T02:37:41.338 INFO:teuthology.orchestra.run.smithi002.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-09-23T02:37:41.338 INFO:teuthology.orchestra.run.smithi002.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-09-23T02:37:41.339 INFO:teuthology.orchestra.run.smithi002.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-09-23T02:37:41.339 INFO:teuthology.orchestra.run.smithi002.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-09-23T02:37:41.350 INFO:teuthology.orchestra.run.smithi002.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-23T02:37:41.350 INFO:teuthology.orchestra.run.smithi002.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-09-23T02:37:41.367 INFO:teuthology.orchestra.run.smithi002.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-09-23T02:37:41.369 INFO:teuthology.orchestra.run.smithi002.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-09-23T02:37:41.369 INFO:teuthology.orchestra.run.smithi002.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-09-23T02:37:41.370 INFO:teuthology.orchestra.run.smithi002.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-09-23T02:37:41.375 INFO:teuthology.orchestra.run.smithi002.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-09-23T02:37:41.376 INFO:teuthology.orchestra.run.smithi002.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-09-23T02:37:41.376 INFO:teuthology.orchestra.run.smithi002.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-09-23T02:37:41.377 INFO:teuthology.orchestra.run.smithi002.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-09-23T02:37:41.384 INFO:teuthology.orchestra.run.smithi002.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-09-23T02:37:41.386 INFO:teuthology.orchestra.run.smithi002.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-09-23T02:37:41.396 INFO:teuthology.orchestra.run.smithi002.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-09-23T02:37:41.397 INFO:teuthology.orchestra.run.smithi002.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-09-23T02:37:41.399 INFO:teuthology.orchestra.run.smithi002.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-09-23T02:37:41.403 INFO:teuthology.orchestra.run.smithi002.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-09-23T02:37:41.404 INFO:teuthology.orchestra.run.smithi002.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-09-23T02:37:41.407 INFO:teuthology.orchestra.run.smithi002.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-09-23T02:37:41.446 INFO:teuthology.orchestra.run.smithi002.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6334-gee696c23-1jammy [3,120 kB] 2023-09-23T02:37:41.460 INFO:teuthology.orchestra.run.smithi002.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-09-23T02:37:41.461 INFO:teuthology.orchestra.run.smithi002.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-09-23T02:37:41.462 INFO:teuthology.orchestra.run.smithi002.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-09-23T02:37:41.463 INFO:teuthology.orchestra.run.smithi002.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-23T02:37:41.464 INFO:teuthology.orchestra.run.smithi002.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-09-23T02:37:41.465 INFO:teuthology.orchestra.run.smithi002.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-09-23T02:37:41.482 INFO:teuthology.orchestra.run.smithi002.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-09-23T02:37:41.484 INFO:teuthology.orchestra.run.smithi002.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-09-23T02:37:41.485 INFO:teuthology.orchestra.run.smithi002.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-09-23T02:37:41.500 INFO:teuthology.orchestra.run.smithi002.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-09-23T02:37:41.500 INFO:teuthology.orchestra.run.smithi002.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-09-23T02:37:41.501 INFO:teuthology.orchestra.run.smithi002.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-09-23T02:37:41.501 INFO:teuthology.orchestra.run.smithi002.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-09-23T02:37:41.501 INFO:teuthology.orchestra.run.smithi002.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-09-23T02:37:41.502 INFO:teuthology.orchestra.run.smithi002.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-09-23T02:37:41.512 INFO:teuthology.orchestra.run.smithi002.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-09-23T02:37:41.513 INFO:teuthology.orchestra.run.smithi002.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-09-23T02:37:41.516 INFO:teuthology.orchestra.run.smithi002.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-09-23T02:37:41.529 INFO:teuthology.orchestra.run.smithi002.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-09-23T02:37:41.530 INFO:teuthology.orchestra.run.smithi002.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-09-23T02:37:41.531 INFO:teuthology.orchestra.run.smithi002.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-09-23T02:37:41.535 INFO:teuthology.orchestra.run.smithi002.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-09-23T02:37:41.537 INFO:teuthology.orchestra.run.smithi002.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-09-23T02:37:41.541 INFO:teuthology.orchestra.run.smithi002.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6334-gee696c23-1jammy [677 kB] 2023-09-23T02:37:41.541 INFO:teuthology.orchestra.run.smithi002.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-09-23T02:37:41.544 INFO:teuthology.orchestra.run.smithi002.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-09-23T02:37:41.546 INFO:teuthology.orchestra.run.smithi002.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-23T02:37:41.546 INFO:teuthology.orchestra.run.smithi002.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-09-23T02:37:41.547 INFO:teuthology.orchestra.run.smithi002.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-09-23T02:37:41.562 INFO:teuthology.orchestra.run.smithi002.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6334-gee696c23-1jammy [345 kB] 2023-09-23T02:37:41.562 INFO:teuthology.orchestra.run.smithi002.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-09-23T02:37:41.563 INFO:teuthology.orchestra.run.smithi002.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-09-23T02:37:41.570 INFO:teuthology.orchestra.run.smithi002.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-09-23T02:37:41.570 INFO:teuthology.orchestra.run.smithi002.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-09-23T02:37:41.571 INFO:teuthology.orchestra.run.smithi002.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-09-23T02:37:41.572 INFO:teuthology.orchestra.run.smithi002.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6334-gee696c23-1jammy [22.9 kB] 2023-09-23T02:37:41.573 INFO:teuthology.orchestra.run.smithi002.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-09-23T02:37:41.573 INFO:teuthology.orchestra.run.smithi002.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6334-gee696c23-1jammy [170 kB] 2023-09-23T02:37:41.574 INFO:teuthology.orchestra.run.smithi002.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-09-23T02:37:41.593 INFO:teuthology.orchestra.run.smithi002.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6334-gee696c23-1jammy [52.4 kB] 2023-09-23T02:37:41.621 INFO:teuthology.orchestra.run.smithi002.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6334-gee696c23-1jammy [318 kB] 2023-09-23T02:37:41.638 INFO:teuthology.orchestra.run.smithi002.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6334-gee696c23-1jammy [6,201 kB] 2023-09-23T02:37:41.849 INFO:teuthology.orchestra.run.smithi002.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6334-gee696c23-1jammy [102 kB] 2023-09-23T02:37:41.881 INFO:teuthology.orchestra.run.smithi002.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6334-gee696c23-1jammy [428 kB] 2023-09-23T02:37:41.943 INFO:teuthology.orchestra.run.smithi002.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6334-gee696c23-1jammy [22.5 MB] 2023-09-23T02:37:42.762 INFO:teuthology.orchestra.run.smithi002.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6334-gee696c23-1jammy [6,841 kB] 2023-09-23T02:37:42.900 INFO:teuthology.orchestra.run.smithi002.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6334-gee696c23-1jammy [228 kB] 2023-09-23T02:37:42.906 INFO:teuthology.orchestra.run.smithi002.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6334-gee696c23-1jammy [117 kB] 2023-09-23T02:37:42.909 INFO:teuthology.orchestra.run.smithi002.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6334-gee696c23-1jammy [1,486 kB] 2023-09-23T02:37:42.959 INFO:teuthology.orchestra.run.smithi002.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6334-gee696c23-1jammy [9,851 kB] 2023-09-23T02:37:43.283 INFO:teuthology.orchestra.run.smithi002.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6334-gee696c23-1jammy [31.7 MB] 2023-09-23T02:37:44.469 INFO:teuthology.orchestra.run.smithi002.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6334-gee696c23-1jammy [3,594 B] 2023-09-23T02:37:44.470 INFO:teuthology.orchestra.run.smithi002.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6334-gee696c23-1jammy [827 kB] 2023-09-23T02:37:44.486 INFO:teuthology.orchestra.run.smithi002.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6334-gee696c23-1jammy [2,135 kB] 2023-09-23T02:37:44.525 INFO:teuthology.orchestra.run.smithi002.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6334-gee696c23-1jammy [251 kB] 2023-09-23T02:37:44.529 INFO:teuthology.orchestra.run.smithi002.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6334-gee696c23-1jammy [128 kB] 2023-09-23T02:37:44.531 INFO:teuthology.orchestra.run.smithi002.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6334-gee696c23-1jammy [1,883 kB] 2023-09-23T02:37:44.554 INFO:teuthology.orchestra.run.smithi002.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6334-gee696c23-1jammy [8,615 kB] 2023-09-23T02:37:44.808 INFO:teuthology.orchestra.run.smithi002.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6334-gee696c23-1jammy [3,724 B] 2023-09-23T02:37:44.809 INFO:teuthology.orchestra.run.smithi002.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6334-gee696c23-1jammy [47.7 MB] 2023-09-23T02:37:46.673 INFO:teuthology.orchestra.run.smithi002.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6334-gee696c23-1jammy [234 kB] 2023-09-23T02:37:46.677 INFO:teuthology.orchestra.run.smithi002.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6334-gee696c23-1jammy [28.3 kB] 2023-09-23T02:37:46.745 INFO:teuthology.orchestra.run.smithi002.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6334-gee696c23-1jammy [12.3 MB] 2023-09-23T02:37:47.031 INFO:teuthology.orchestra.run.smithi002.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6334-gee696c23-1jammy [80.6 kB] 2023-09-23T02:37:47.571 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 183 MB in 6s (30.7 MB/s) 2023-09-23T02:37:47.703 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libfmt8:amd64. 2023-09-23T02:37:47.780 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 166009 files and directories currently installed.) 2023-09-23T02:37:47.786 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-09-23T02:37:47.869 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-23T02:37:48.147 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-09-23T02:37:48.174 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-09-23T02:37:48.205 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-23T02:37:48.534 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-23T02:37:48.560 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-09-23T02:37:48.600 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-23T02:37:48.936 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-23T02:37:48.963 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-09-23T02:37:49.003 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-23T02:37:49.423 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-23T02:37:49.450 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-23T02:37:49.498 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:37:49.884 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-23T02:37:49.911 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-23T02:37:49.951 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:37:50.296 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-23T02:37:50.322 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-09-23T02:37:50.362 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:37:50.707 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-09-23T02:37:50.734 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-09-23T02:37:50.774 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-23T02:37:51.093 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librados2. 2023-09-23T02:37:51.120 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../008-librados2_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:51.168 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librados2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:51.621 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librbd1. 2023-09-23T02:37:51.647 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../009-librbd1_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librbd1 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:52.065 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs2. 2023-09-23T02:37:52.092 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:52.123 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:52.401 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rados. 2023-09-23T02:37:52.417 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:52.459 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rados (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:52.820 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-23T02:37:52.847 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:37:52.886 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-argparse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:53.139 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cephfs. 2023-09-23T02:37:53.166 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:53.206 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cephfs (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:53.525 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-23T02:37:53.541 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:37:53.583 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-ceph-common (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:53.887 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-23T02:37:53.913 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-09-23T02:37:53.953 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-23T02:37:54.248 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-prettytable. 2023-09-23T02:37:54.274 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-09-23T02:37:54.314 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-09-23T02:37:54.892 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rbd. 2023-09-23T02:37:54.919 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:54.958 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rbd (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:55.344 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-23T02:37:55.371 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-09-23T02:37:55.411 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-23T02:37:55.714 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package librgw2. 2023-09-23T02:37:55.741 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../019-librgw2_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:55.772 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking librgw2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:56.241 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rgw. 2023-09-23T02:37:56.257 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:56.299 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rgw (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:56.653 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package liboath0:amd64. 2023-09-23T02:37:56.679 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-09-23T02:37:56.718 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-09-23T02:37:57.030 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libradosstriper1. 2023-09-23T02:37:57.057 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:57.096 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libradosstriper1 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:57.433 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-common. 2023-09-23T02:37:57.460 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:57.491 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-common (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:58.797 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-base. 2023-09-23T02:37:58.824 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:37:58.881 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-base (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:37:59.551 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-09-23T02:37:59.728 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-09-23T02:37:59.728 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-09-23T02:38:00.053 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cheroot. 2023-09-23T02:38:00.080 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-09-23T02:38:00.120 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-23T02:38:00.431 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-09-23T02:38:00.458 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-09-23T02:38:00.498 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-09-23T02:38:00.817 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.text. 2023-09-23T02:38:00.844 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-09-23T02:38:00.884 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-09-23T02:38:01.128 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-09-23T02:38:01.144 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-09-23T02:38:01.178 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-09-23T02:38:01.439 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempora. 2023-09-23T02:38:01.466 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-09-23T02:38:01.506 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-09-23T02:38:01.809 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-portend. 2023-09-23T02:38:01.835 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-09-23T02:38:01.875 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-09-23T02:38:02.161 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-09-23T02:38:02.188 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-09-23T02:38:02.228 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-09-23T02:38:02.472 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-23T02:38:02.499 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-09-23T02:38:02.539 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-09-23T02:38:03.026 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-natsort. 2023-09-23T02:38:03.053 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-09-23T02:38:03.092 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-09-23T02:38:03.378 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-logutils. 2023-09-23T02:38:03.405 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-09-23T02:38:03.445 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-09-23T02:38:03.715 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-23T02:38:03.741 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-09-23T02:38:03.781 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-09-23T02:38:04.067 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-mako. 2023-09-23T02:38:04.094 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-09-23T02:38:04.134 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-23T02:38:04.470 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-23T02:38:04.497 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-23T02:38:04.537 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-23T02:38:04.856 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-23T02:38:04.883 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-09-23T02:38:04.923 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-09-23T02:38:05.150 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webob. 2023-09-23T02:38:05.166 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-09-23T02:38:05.200 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-09-23T02:38:05.520 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-waitress. 2023-09-23T02:38:05.547 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-09-23T02:38:05.590 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-23T02:38:05.898 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-tempita. 2023-09-23T02:38:05.925 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-09-23T02:38:05.964 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-09-23T02:38:06.259 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-paste. 2023-09-23T02:38:06.286 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-09-23T02:38:06.325 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-09-23T02:38:06.637 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-23T02:38:06.664 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-09-23T02:38:06.695 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-09-23T02:38:06.958 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-23T02:38:06.982 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-09-23T02:38:07.025 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-09-23T02:38:07.328 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-webtest. 2023-09-23T02:38:07.355 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-09-23T02:38:07.394 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-09-23T02:38:07.730 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pecan. 2023-09-23T02:38:07.757 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-09-23T02:38:07.797 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-09-23T02:38:08.135 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-certifi. 2023-09-23T02:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-09-23T02:38:08.201 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-09-23T02:38:08.538 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-idna. 2023-09-23T02:38:08.564 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-09-23T02:38:08.604 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-idna (3.3-1) ... 2023-09-23T02:38:08.874 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-urllib3. 2023-09-23T02:38:08.889 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-09-23T02:38:08.923 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-09-23T02:38:09.202 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests. 2023-09-23T02:38:09.218 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-09-23T02:38:09.251 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-23T02:38:09.563 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-23T02:38:09.589 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-09-23T02:38:09.629 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-23T02:38:10.032 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-23T02:38:10.059 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:10.107 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:10.487 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-23T02:38:10.514 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:10.553 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:10.839 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-openssl. 2023-09-23T02:38:10.866 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-09-23T02:38:10.897 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-09-23T02:38:11.226 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr. 2023-09-23T02:38:11.253 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:11.292 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:11.704 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mon. 2023-09-23T02:38:11.731 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:11.770 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mon (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:12.641 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-osd. 2023-09-23T02:38:12.667 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:12.707 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-osd (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:15.077 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph. 2023-09-23T02:38:15.104 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../059-ceph_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:15.144 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:15.446 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-fuse. 2023-09-23T02:38:15.462 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:15.496 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-fuse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:15.883 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mds. 2023-09-23T02:38:15.910 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:15.949 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mds (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:16.327 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package cephadm. 2023-09-23T02:38:16.354 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../062-cephadm_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:16.394 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking cephadm (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:16.663 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-asyncssh. 2023-09-23T02:38:16.690 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-09-23T02:38:16.721 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-09-23T02:38:17.032 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-babel-localedata. 2023-09-23T02:38:17.060 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-09-23T02:38:17.099 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-23T02:38:18.154 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-babel. 2023-09-23T02:38:18.182 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-09-23T02:38:18.220 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-09-23T02:38:18.548 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-jinja2. 2023-09-23T02:38:18.576 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-09-23T02:38:18.615 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-09-23T02:38:18.969 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-23T02:38:18.996 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:19.035 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:19.355 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-23T02:38:19.382 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-09-23T02:38:19.421 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-09-23T02:38:19.666 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-23T02:38:19.684 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-09-23T02:38:19.716 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-23T02:38:20.011 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-routes. 2023-09-23T02:38:20.038 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-09-23T02:38:20.069 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-09-23T02:38:20.614 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-23T02:38:20.714 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:20.714 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:21.126 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-09-23T02:38:21.153 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-09-23T02:38:21.192 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-23T02:38:21.679 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-joblib. 2023-09-23T02:38:21.706 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-09-23T02:38:21.745 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-09-23T02:38:22.057 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-09-23T02:38:22.084 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-09-23T02:38:22.115 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-09-23T02:38:22.460 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-sklearn. 2023-09-23T02:38:22.487 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-09-23T02:38:22.526 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-23T02:38:23.227 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-23T02:38:23.255 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:23.301 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:24.246 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-cachetools. 2023-09-23T02:38:24.273 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-09-23T02:38:24.312 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-09-23T02:38:24.615 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-23T02:38:24.643 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-09-23T02:38:24.682 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-09-23T02:38:25.010 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-23T02:38:25.038 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-09-23T02:38:25.068 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-09-23T02:38:25.347 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-rsa. 2023-09-23T02:38:25.374 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-09-23T02:38:25.404 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-rsa (4.8-1) ... 2023-09-23T02:38:25.716 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-google-auth. 2023-09-23T02:38:25.744 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-09-23T02:38:25.783 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-09-23T02:38:26.103 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-23T02:38:26.130 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-09-23T02:38:26.169 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-23T02:38:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-websocket. 2023-09-23T02:38:26.584 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-09-23T02:38:26.623 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-09-23T02:38:26.910 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-23T02:38:26.938 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-09-23T02:38:26.977 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-23T02:38:27.590 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-23T02:38:27.607 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:27.648 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:27.943 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-23T02:38:27.971 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-09-23T02:38:28.010 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-23T02:38:28.371 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-23T02:38:28.399 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-09-23T02:38:28.437 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-23T02:38:28.707 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package jq. 2023-09-23T02:38:28.723 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-09-23T02:38:28.765 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-09-23T02:38:29.001 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package socat. 2023-09-23T02:38:29.029 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-09-23T02:38:29.059 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-09-23T02:38:29.354 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package xmlstarlet. 2023-09-23T02:38:29.382 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-09-23T02:38:29.420 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-09-23T02:38:29.775 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-test. 2023-09-23T02:38:29.791 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:29.824 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-test (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:32.262 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package ceph-volume. 2023-09-23T02:38:32.290 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6334-gee696c23-1jammy_all.deb ... 2023-09-23T02:38:32.328 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking ceph-volume (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:32.666 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-23T02:38:32.694 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:32.733 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libcephfs-dev (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:33.095 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package nvme-cli. 2023-09-23T02:38:33.122 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-09-23T02:38:33.161 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-09-23T02:38:33.516 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-09-23T02:38:33.544 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-09-23T02:38:33.582 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-09-23T02:38:34.002 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-iniconfig. 2023-09-23T02:38:34.030 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-09-23T02:38:34.069 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-09-23T02:38:34.322 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pastescript. 2023-09-23T02:38:34.349 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-09-23T02:38:34.379 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-23T02:38:34.641 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pluggy. 2023-09-23T02:38:34.669 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-09-23T02:38:34.699 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-09-23T02:38:35.011 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-psutil. 2023-09-23T02:38:35.039 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-09-23T02:38:35.077 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-09-23T02:38:35.364 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-py. 2023-09-23T02:38:35.392 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-09-23T02:38:35.421 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-py (1.10.0-1) ... 2023-09-23T02:38:35.700 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pygments. 2023-09-23T02:38:35.728 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-09-23T02:38:35.866 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-09-23T02:38:36.346 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-23T02:38:36.374 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-09-23T02:38:36.412 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-09-23T02:38:36.715 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-toml. 2023-09-23T02:38:36.743 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-09-23T02:38:36.781 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-09-23T02:38:37.084 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-pytest. 2023-09-23T02:38:37.112 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-09-23T02:38:37.159 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-09-23T02:38:37.479 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package python3-simplejson. 2023-09-23T02:38:37.507 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-09-23T02:38:37.545 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-09-23T02:38:37.814 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-23T02:38:37.831 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-09-23T02:38:37.872 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-09-23T02:38:38.627 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package radosgw. 2023-09-23T02:38:38.655 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../107-radosgw_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:38.701 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking radosgw (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:39.621 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package rbd-fuse. 2023-09-23T02:38:39.638 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6334-gee696c23-1jammy_amd64.deb ... 2023-09-23T02:38:39.679 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking rbd-fuse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:39.984 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-09-23T02:38:40.234 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-09-23T02:38:40.368 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-09-23T02:38:40.648 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-09-23T02:38:41.007 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-09-23T02:38:41.495 INFO:teuthology.orchestra.run.smithi002.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-09-23T02:38:41.853 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up cephadm (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:42.581 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user cephadm....done 2023-09-23T02:38:43.067 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-09-23T02:38:43.362 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-09-23T02:38:43.622 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-09-23T02:38:43.748 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-09-23T02:38:44.008 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-23T02:38:44.284 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-09-23T02:38:44.460 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-py (1.10.0-1) ... 2023-09-23T02:38:44.811 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-09-23T02:38:45.180 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-09-23T02:38:45.439 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-09-23T02:38:45.790 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-09-23T02:38:46.134 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-09-23T02:38:46.418 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-09-23T02:38:46.711 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-argparse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:46.978 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-09-23T02:38:47.121 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-09-23T02:38:47.497 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-09-23T02:38:47.623 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-09-23T02:38:48.000 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-09-23T02:38:48.126 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-09-23T02:38:48.410 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-09-23T02:38:48.686 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-09-23T02:38:49.004 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-09-23T02:38:49.731 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-09-23T02:38:50.007 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-09-23T02:38:50.133 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-09-23T02:38:50.269 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-09-23T02:38:50.579 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-09-23T02:38:50.980 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-09-23T02:38:51.239 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-09-23T02:38:51.616 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-09-23T02:38:51.909 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-09-23T02:38:52.277 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-09-23T02:38:52.536 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-09-23T02:38:52.687 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-ceph-common (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:38:52.988 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-idna (3.3-1) ... 2023-09-23T02:38:53.315 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-09-23T02:38:54.579 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:38:54.713 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-toml (0.10.2-1) ... 2023-09-23T02:38:54.973 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-09-23T02:38:55.099 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-09-23T02:38:55.225 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-09-23T02:38:55.534 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-09-23T02:38:55.811 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-09-23T02:38:56.062 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:38:56.188 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-09-23T02:38:56.314 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-09-23T02:38:56.632 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-09-23T02:38:56.916 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-09-23T02:38:57.176 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-09-23T02:38:57.694 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-23T02:38:57.945 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-09-23T02:38:58.221 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-09-23T02:38:58.347 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-09-23T02:38:58.640 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-09-23T02:38:58.766 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-09-23T02:38:59.067 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-09-23T02:38:59.385 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-09-23T02:38:59.703 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-09-23T02:39:00.110 INFO:teuthology.orchestra.run.smithi002.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-09-23T02:39:00.356 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-09-23T02:39:00.482 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-09-23T02:39:00.917 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-09-23T02:39:01.217 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-portend (3.0.0-1) ... 2023-09-23T02:39:01.494 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-09-23T02:39:01.620 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-09-23T02:39:01.963 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-09-23T02:39:02.272 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-09-23T02:39:02.565 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-09-23T02:39:03.013 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rsa (4.8-1) ... 2023-09-23T02:39:03.159 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-09-23T02:39:03.443 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-09-23T02:39:03.711 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-09-23T02:39:04.112 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-23T02:39:04.405 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-09-23T02:39:04.774 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-09-23T02:39:04.900 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librados2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:05.060 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librgw2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:05.202 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:05.328 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs2 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:05.454 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libradosstriper1 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:05.666 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-09-23T02:39:05.942 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up librbd1 (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:06.093 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-modules-core (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:06.236 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-fuse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:06.452 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-23T02:39:06.452 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-23T02:39:06.996 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libcephfs-dev (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:07.122 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rados (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:07.248 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-09-23T02:39:08.519 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rbd (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:08.645 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up rbd-fuse (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:08.771 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-rgw (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:08.897 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up python3-cephfs (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:09.040 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-common (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:09.600 INFO:teuthology.orchestra.run.smithi002.stdout:Adding group ceph....done 2023-09-23T02:39:09.901 INFO:teuthology.orchestra.run.smithi002.stdout:Adding system user ceph....done 2023-09-23T02:39:10.044 INFO:teuthology.orchestra.run.smithi002.stdout:Setting system user ceph properties....done 2023-09-23T02:39:10.063 INFO:teuthology.orchestra.run.smithi002.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-23T02:39:10.206 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-23T02:39:10.510 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-23T02:39:11.041 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-test (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:11.167 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up radosgw (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:11.440 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-23T02:39:11.441 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-23T02:39:12.253 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-base (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:12.609 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-23T02:39:13.154 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mds (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-23T02:39:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-23T02:39:13.889 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:14.116 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-23T02:39:14.117 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-23T02:39:14.640 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-osd (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:14.970 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-23T02:39:14.970 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-23T02:39:15.492 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:15.619 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:15.770 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mon (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:15.967 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-23T02:39:15.967 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-23T02:39:16.550 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-cephadm (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:16.754 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:17.013 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-mgr-dashboard (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:17.264 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up ceph-volume (18.0.0-6334-gee696c23-1jammy) ... 2023-09-23T02:39:17.451 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-09-23T02:39:18.151 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-09-23T02:39:18.624 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-23T02:39:18.627 DEBUG:teuthology.parallel:result is None 2023-09-23T02:39:18.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:39:18.769 DEBUG:teuthology.orchestra.run.smithi002:> dpkg-query -W -f '${Version}' ceph 2023-09-23T02:39:18.788 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-6334-gee696c23-1jammy 2023-09-23T02:39:18.789 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6334-gee696c23-1jammy 2023-09-23T02:39:18.789 INFO:teuthology.task.install:The correct ceph version 18.0.0-6334-gee696c23-1jammy is installed. 2023-09-23T02:39:18.789 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-23T02:39:18.789 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:18.789 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-23T02:39:18.844 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-23T02:39:18.844 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:18.844 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2023-09-23T02:39:18.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-23T02:39:18.956 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-23T02:39:18.956 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:18.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-23T02:39:19.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-23T02:39:19.068 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-23T02:39:19.068 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:19.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2023-09-23T02:39:19.124 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-23T02:39:19.180 INFO:teuthology.run_tasks:Running task ceph... 2023-09-23T02:39:19.242 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-23T02:39:19.242 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2023-09-23T02:39:19.255 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-23T02:39:19.255 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-23T02:39:19.308 INFO:tasks.ceph:Creating extra log directories... 2023-09-23T02:39:19.308 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-23T02:39:19.371 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-23T02:39:19.371 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-23T02:39:19.371 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437', 'branch': 'wip-62891', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7401437', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-62891'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 61}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)'], 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-62891', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 3804, 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-62891', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e', 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTHhvD+lHCTnuCMxkFMRGgP2c+8SiXjzcY+NkKI2aVnWsEVb0yz3qCxaMsEyq5Or+5R29zNh9dNSgt0PSx0Ozc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2', 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile', 'sudo ceph osd pool set datapool allow_ec_overwrites true', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-23_02:20:47', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-23T02:39:19.372 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-23T02:39:19.413 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2023-09-23T02:39:19.469 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:19.469 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2023-09-23T02:39:19.517 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-23T02:39:19.517 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2023-09-23T02:39:19.565 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-23T02:39:19.565 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T02:39:19.565 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 877 Links: 1 2023-09-23T02:39:19.565 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T02:39:19.566 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-09-23 02:34:11.750546118 +0000 2023-09-23T02:39:19.566 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-09-23 02:34:11.746546154 +0000 2023-09-23T02:39:19.566 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-09-23 02:34:11.746546154 +0000 2023-09-23T02:39:19.566 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-09-23T02:39:19.566 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-23T02:39:19.619 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-09-23T02:39:19.619 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-09-23T02:39:19.619 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000310744 s, 1.6 MB/s 2023-09-23T02:39:19.620 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-23T02:39:19.670 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2023-09-23T02:39:19.717 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-23T02:39:19.717 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-09-23 02:34:12.654538043 +0000 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-09-23 02:34:12.318541044 +0000 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-09-23 02:34:12.318541044 +0000 2023-09-23T02:39:19.718 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-09-23T02:39:19.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-23T02:39:19.772 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-09-23T02:39:19.772 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-09-23T02:39:19.772 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00028336 s, 1.8 MB/s 2023-09-23T02:39:19.773 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-23T02:39:19.822 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2023-09-23T02:39:19.869 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-23T02:39:19.869 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T02:39:19.869 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 919 Links: 1 2023-09-23T02:39:19.870 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T02:39:19.870 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-09-23 02:34:13.242532791 +0000 2023-09-23T02:39:19.870 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-09-23 02:34:12.914535721 +0000 2023-09-23T02:39:19.870 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-09-23 02:34:12.914535721 +0000 2023-09-23T02:39:19.870 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-09-23T02:39:19.870 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-23T02:39:19.923 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-09-23T02:39:19.924 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-09-23T02:39:19.924 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000324921 s, 1.6 MB/s 2023-09-23T02:39:19.925 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-23T02:39:19.974 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2023-09-23T02:39:20.021 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-23T02:39:20.021 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-23T02:39:20.021 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-09-23T02:39:20.021 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-23T02:39:20.021 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-09-23 02:34:13.830527539 +0000 2023-09-23T02:39:20.022 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-09-23 02:34:13.470530755 +0000 2023-09-23T02:39:20.022 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-09-23 02:34:13.470530755 +0000 2023-09-23T02:39:20.022 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: - 2023-09-23T02:39:20.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-23T02:39:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-09-23T02:39:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-09-23T02:39:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000201835 s, 2.5 MB/s 2023-09-23T02:39:20.076 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-23T02:39:20.126 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T02:39:20.126 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-23T02:39:20.126 INFO:tasks.ceph:Generating config... 2023-09-23T02:39:20.127 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-09-23T02:39:20.128 INFO:tasks.ceph:[client] rbd default features = 61 2023-09-23T02:39:20.128 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-23T02:39:20.128 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-09-23T02:39:20.128 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-23T02:39:20.128 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-23T02:39:20.128 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-23T02:39:20.128 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-09-23T02:39:20.128 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-23T02:39:20.129 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-23T02:39:20.129 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-09-23T02:39:20.129 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] enable experimental unrecoverable data corrupting features = * 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-23T02:39:20.130 INFO:tasks.ceph:[osd] osd debug randomize hobject sort order = False 2023-09-23T02:39:20.131 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-23T02:39:20.131 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-23T02:39:20.131 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-23T02:39:20.131 INFO:tasks.ceph:Setting up mon.a... 2023-09-23T02:39:20.131 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-23T02:39:20.201 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2023-09-23T02:39:20.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-23T02:39:20.292 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-23T02:39:20.349 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2')] 2023-09-23T02:39:20.350 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'enable experimental unrecoverable data corrupting features': '*', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug randomize hobject sort order': False, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 61}, 'mon.a': {}} 2023-09-23T02:39:20.351 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:20.351 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-23T02:39:20.397 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:20.476 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:20.476 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid f964ea34-ba1c-42d5-92d6-b55c00c696eb 2023-09-23T02:39:20.476 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:fsid f964ea34-ba1c-42d5-92d6-b55c00c696eb 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2023-09-23T02:39:20.479476+0000 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-09-23T02:39:20.479476+0000 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 16 (pacific) 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 1 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2023-09-23T02:39:20.477 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-09-23T02:39:20.479 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-23T02:39:20.525 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f964ea34-ba1c-42d5-92d6-b55c00c696eb... 2023-09-23T02:39:20.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.599 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2023-09-23T02:39:20.600 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2023-09-23T02:39:20.601 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2023-09-23T02:39:20.602 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.603 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = 172.21.15.2 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: mon client directed command retry = 5 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on pool no app = False 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: ms inject socket failures = 5000 2023-09-23T02:39:20.604 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = f964ea34-ba1c-42d5-92d6-b55c00c696eb 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.605 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2023-09-23T02:39:20.606 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.607 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bdev async discard = True 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bdev enable discard = True 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore allocator = bitmap 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore block size = 96636764160 2023-09-23T02:39:20.608 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore fsck on mount = True 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluefs = 1/20 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluestore = 1/20 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: debug rocksdb = 4/10 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: enable experimental unrecoverable data corrupting features = * 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd backfillfull_ratio = 0.85 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd full ratio = 0.9 2023-09-23T02:39:20.609 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd nearfull ratio = 0.8 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug randomize hobject sort order = False 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: osd failsafe full ratio = 0.95 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: osd objectstore = bluestore 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: osd shutdown pgref assert = True 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2023-09-23T02:39:20.610 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: debug rbd = 20 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2023-09-23T02:39:20.611 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2023-09-23T02:39:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.613 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2023-09-23T02:39:20.614 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2023-09-23T02:39:20.615 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-23T02:39:20.615 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-23T02:39:20.615 INFO:teuthology.orchestra.run.smithi002.stdout: rbd default data pool = datapool 2023-09-23T02:39:20.615 INFO:teuthology.orchestra.run.smithi002.stdout: rbd default features = 61 2023-09-23T02:39:20.615 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2023-09-23T02:39:20.616 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-23T02:39:20.616 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-23T02:39:20.708 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-23T02:39:20.708 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:20.708 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-23T02:39:20.761 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:20.762 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-23T02:39:20.809 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:39:20.809 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:20.809 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-23T02:39:20.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-23T02:39:20.874 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:20.875 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:20.921 INFO:tasks.ceph:Setting up mon nodes... 2023-09-23T02:39:20.922 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-23T02:39:20.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-23T02:39:21.008 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-23T02:39:21.011 INFO:tasks.ceph:Setting up mds nodes... 2023-09-23T02:39:21.012 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-23T02:39:21.012 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-23T02:39:21.046 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-23T02:39:21.056 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-23T02:39:21.056 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-23T02:39:21.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-23T02:39:21.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T02:39:21.113 INFO:tasks.ceph:role: osd.0 2023-09-23T02:39:21.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:39:21.114 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-23T02:39:21.168 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T02:39:21.169 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T02:39:21.170 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T02:39:21.170 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T02:39:21.192 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-09-23T02:39:21.194 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-09-23T02:39:21.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-23T02:39:21.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-23T02:39:21.305 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-09-23T02:39:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T02:39:21.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-23T02:39:21.358 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T02:39:21.358 INFO:tasks.ceph:role: osd.1 2023-09-23T02:39:21.358 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:39:21.358 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-23T02:39:21.411 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T02:39:21.412 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T02:39:21.435 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-09-23T02:39:21.437 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-09-23T02:39:21.437 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-23T02:39:21.497 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-23T02:39:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T02:39:21.553 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-09-23T02:39:21.553 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-23T02:39:21.605 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-23T02:39:21.605 INFO:tasks.ceph:role: osd.2 2023-09-23T02:39:21.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:39:21.606 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-23T02:39:21.661 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-23T02:39:21.685 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2023-09-23T02:39:21.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2023-09-23T02:39:21.687 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-23T02:39:21.744 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-23T02:39:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T02:39:21.797 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: /sbin/restorecon: command not found 2023-09-23T02:39:21.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:21.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:21.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:21.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:21.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-23T02:39:21.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-23T02:39:21.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T02:39:21.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T02:39:21.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-23T02:39:21.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:21.907+0000 7fecbfe45640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-23T02:39:24.937 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-23T02:39:24.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:25.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.031+0000 7f868f6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:25.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.031+0000 7f868f6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:25.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.031+0000 7f868f6e2640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:25.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-23T02:39:25.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-23T02:39:25.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T02:39:25.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T02:39:25.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-23T02:39:25.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:25.035+0000 7f868f6e2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-23T02:39:28.041 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-23T02:39:28.097 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:28.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.179+0000 7efc4a235640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:28.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.179+0000 7efc4a235640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:28.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:28.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-23T02:39:28.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-23T02:39:28.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T02:39:28.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T02:39:28.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-23T02:39:28.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-09-23T02:39:28.183+0000 7efc4a235640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-23T02:39:31.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-23T02:39:31.245 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-23T02:39:31.245 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:31.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-23T02:39:31.300 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:31.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-23T02:39:31.356 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:31.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-23T02:39:31.411 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:31.411 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-23T02:39:31.468 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:31.469 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-23T02:39:31.516 INFO:tasks.ceph:Adding keys to all mons... 2023-09-23T02:39:31.516 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-23T02:39:31.570 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDZTw5lTDO0ABAAj52bjC+pqBg53Il8LDHnOA== 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDZTw5lx+k8NhAAFq0gndnIKc61jSNr0txDog== 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDdTw5l2YghAhAA+Htehy4k3PFwf3gnqhOcNA== 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDgTw5lBMcDCxAAerINDBYSsnHtqZQKA8h1MQ== 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2023-09-23T02:39:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDZTw5lExnvAhAADZqHPR4SjpKAzuaHWK+thg== 2023-09-23T02:39:31.572 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-23T02:39:31.663 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T02:39:31.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T02:39:31.793 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-23T02:39:31.832 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-23T02:39:31.925 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-23T02:39:31.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-23T02:39:31.937 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-23T02:39:32.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-23T02:39:32.043 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-23T02:39:32.088 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-23T02:39:32.088 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-23T02:39:32.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-23T02:39:32.133 INFO:tasks.ceph.mon.a:Started 2023-09-23T02:39:32.134 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-23T02:39:32.134 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-23T02:39:32.134 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-23T02:39:32.136 INFO:tasks.ceph.mgr.x:Started 2023-09-23T02:39:32.137 INFO:tasks.ceph:Setting crush tunables to default 2023-09-23T02:39:32.137 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2023-09-23T02:39:32.382 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2023-09-23T02:39:32.400 INFO:tasks.ceph:check_enable_crimson: False 2023-09-23T02:39:32.400 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-23T02:39:32.400 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:32.400 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-23T02:39:32.410 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:32.410 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-23T02:39:32.463 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:39:32.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-23T02:39:32.503 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:32.503+0000 7f39faace180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-23T02:39:32.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 274b1544-bb02-4626-b8d0-a991f832cfa9 0 2023-09-23T02:39:32.760 INFO:teuthology.orchestra.run.smithi002.stdout:0 2023-09-23T02:39:32.779 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new dff4c364-8a65-4fdb-b791-b4707748ec65 1 2023-09-23T02:39:32.977 INFO:teuthology.orchestra.run.smithi002.stdout:1 2023-09-23T02:39:32.993 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd new 1f73dc35-18f2-49cc-af51-be0c48ebdc9d 2 2023-09-23T02:39:33.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:33.035+0000 7f39faace180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-23T02:39:33.182 INFO:teuthology.orchestra.run.smithi002.stdout:2 2023-09-23T02:39:33.198 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-23T02:39:33.199 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-23T02:39:33.226 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:33.227+0000 7f39faace180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-23T02:39:33.228 INFO:tasks.ceph.osd.0:Started 2023-09-23T02:39:33.228 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-23T02:39:33.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-23T02:39:33.231 INFO:tasks.ceph.osd.1:Started 2023-09-23T02:39:33.232 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-23T02:39:33.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-23T02:39:33.235 INFO:tasks.ceph.osd.2:Started 2023-09-23T02:39:33.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:33.281 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:33.283+0000 7f6e0eac7640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.287 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:33.287+0000 7f6e0eac7640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.287 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:33.287+0000 7f6e0eac7640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.290 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:33.291+0000 7fa3a5d59640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.295 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:33.295+0000 7fa3a5d59640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.295 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:33.295+0000 7fa3a5d59640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.307 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:33.307+0000 7f3c5dacf640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.312 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:33.311+0000 7f3c5dacf640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.312 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:33.311+0000 7f3c5dacf640 -1 WARNING: all dangerous and experimental features are enabled. 2023-09-23T02:39:33.414 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:33.415 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:33.183919+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:33.430 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-23T02:39:33.430 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-23T02:39:33.732 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:33.851 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:33.851+0000 7f39faace180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-23T02:39:33.853 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:33.855+0000 7f6e0eac7640 -1 Falling back to public interface 2023-09-23T02:39:33.860 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:33.863+0000 7fa3a5d59640 -1 Falling back to public interface 2023-09-23T02:39:33.868 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:33.871+0000 7f3c5dacf640 -1 Falling back to public interface 2023-09-23T02:39:33.911 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:33.911+0000 7f39faace180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-23T02:39:33.921 INFO:teuthology.misc.health.smithi002.stdout: 2023-09-23T02:39:33.921 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":5,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:33.183919+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:33.936 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-23T02:39:33.967 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:33.967+0000 7f39faace180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-23T02:39:34.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.047+0000 7f39faace180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-23T02:39:34.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.103+0000 7f39faace180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-23T02:39:34.180 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.183+0000 7f39faace180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-23T02:39:34.234 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.235+0000 7f39faace180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-23T02:39:34.380 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.379+0000 7f39faace180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-23T02:39:34.442 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:34.443+0000 7f39faace180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-23T02:39:35.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-09-23T02:39:35.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-09-23T02:39:35.055 INFO:tasks.ceph.mgr.x.smithi002.stderr: from numpy import show_config as show_numpy_config 2023-09-23T02:39:35.116 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:35.119+0000 7f39faace180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-23T02:39:35.289 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:35.291+0000 7f39faace180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-23T02:39:35.468 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:35.471+0000 7f39faace180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-23T02:39:35.534 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:35.535+0000 7f39faace180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-23T02:39:35.663 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:35.663+0000 7f39faace180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-23T02:39:36.292 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:36.295+0000 7f39faace180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-23T02:39:36.370 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:36.371+0000 7f39faace180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-23T02:39:36.556 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:36.555+0000 7f39faace180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-23T02:39:36.635 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:36.635+0000 7f39faace180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-23T02:39:36.942 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:36.943+0000 7f39faace180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-23T02:39:37.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:37.015+0000 7f39faace180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-23T02:39:37.231 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:37.231+0000 7fa3a5d59640 -1 osd.1 0 log_to_monitors true 2023-09-23T02:39:37.728 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:37.731+0000 7f6e0eac7640 -1 osd.0 0 log_to_monitors true 2023-09-23T02:39:37.772 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:37.775+0000 7f3c5dacf640 -1 osd.2 0 log_to_monitors true 2023-09-23T02:39:38.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:39:38.051+0000 7f39a2242640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-23T02:39:40.238 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:40.582 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:40.583+0000 7fa3a2d16640 -1 osd.1 0 waiting for initial osdmap 2023-09-23T02:39:40.600 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:39:40.603+0000 7fa399b25640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T02:39:40.623 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:40.623+0000 7f6e0a24c640 -1 osd.0 0 waiting for initial osdmap 2023-09-23T02:39:40.656 INFO:tasks.ceph.osd.0.smithi002.stderr:2023-09-23T02:39:40.659+0000 7f6e02893640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T02:39:40.657 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:40.659+0000 7f3c59254640 -1 osd.2 0 waiting for initial osdmap 2023-09-23T02:39:40.671 INFO:teuthology.misc.health.smithi002.stdout: 2023-09-23T02:39:40.671 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":7,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:39.215732+0000","last_up_change":"0.000000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:40.686 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-23T02:39:40.696 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:39:40.699+0000 7f3c5189b640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-23T02:39:46.989 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:47.287 INFO:teuthology.misc.health.smithi002.stdout: 2023-09-23T02:39:47.287 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":12,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:45.598040+0000","last_up_change":"2023-09-23T02:39:41.587263+0000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T02:39:43.121371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6801","nonce":474982742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6803","nonce":474982742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6807","nonce":474982742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6805","nonce":474982742}]},"public_addr":"172.21.15.2:6801/474982742","cluster_addr":"172.21.15.2:6803/474982742","heartbeat_back_addr":"172.21.15.2:6807/474982742","heartbeat_front_addr":"172.21.15.2:6805/474982742","state":["exists","up"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6809","nonce":2376239628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6811","nonce":2376239628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6815","nonce":2376239628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6813","nonce":2376239628}]},"public_addr":"172.21.15.2:6809/2376239628","cluster_addr":"172.21.15.2:6811/2376239628","heartbeat_back_addr":"172.21.15.2:6815/2376239628","heartbeat_front_addr":"172.21.15.2:6813/2376239628","state":["exists","up"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6817","nonce":363956679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6819","nonce":363956679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6823","nonce":363956679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6821","nonce":363956679}]},"public_addr":"172.21.15.2:6817/363956679","cluster_addr":"172.21.15.2:6819/363956679","heartbeat_back_addr":"172.21.15.2:6823/363956679","heartbeat_front_addr":"172.21.15.2:6821/363956679","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.767365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.221094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.758039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:47.304 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-23T02:39:47.304 INFO:tasks.ceph:Creating RBD pool 2023-09-23T02:39:47.305 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-23T02:39:48.216 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'rbd' created 2023-09-23T02:39:48.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-23T02:39:49.228 INFO:teuthology.orchestra.run.smithi002.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-23T02:39:49.262 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-23T02:39:49.263 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-23T02:39:49.263 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-23T02:39:49.263 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-23T02:39:49.263 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-23T02:39:49.597 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:49.614 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3422404231},{"type":"v1","addr":"172.21.15.2:6825","nonce":3422404231}]},"active_addr":"172.21.15.2:6825/3422404231","active_change":"2023-09-23T02:39:37.037422+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3434097723}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3929330339}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":726649277}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":62844154}]}]} 2023-09-23T02:39:49.617 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-23T02:39:49.617 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-23T02:39:49.617 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:49.913 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:49.913 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":14,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:49.225833+0000","last_up_change":"2023-09-23T02:39:41.587263+0000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T02:39:43.121371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T02:39:47.647096+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6801","nonce":474982742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6803","nonce":474982742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6807","nonce":474982742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6805","nonce":474982742}]},"public_addr":"172.21.15.2:6801/474982742","cluster_addr":"172.21.15.2:6803/474982742","heartbeat_back_addr":"172.21.15.2:6807/474982742","heartbeat_front_addr":"172.21.15.2:6805/474982742","state":["exists","up"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6809","nonce":2376239628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6811","nonce":2376239628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6815","nonce":2376239628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6813","nonce":2376239628}]},"public_addr":"172.21.15.2:6809/2376239628","cluster_addr":"172.21.15.2:6811/2376239628","heartbeat_back_addr":"172.21.15.2:6815/2376239628","heartbeat_front_addr":"172.21.15.2:6813/2376239628","state":["exists","up"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6817","nonce":363956679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6819","nonce":363956679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6823","nonce":363956679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6821","nonce":363956679}]},"public_addr":"172.21.15.2:6817/363956679","cluster_addr":"172.21.15.2:6819/363956679","heartbeat_back_addr":"172.21.15.2:6823/363956679","heartbeat_front_addr":"172.21.15.2:6821/363956679","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.767365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.221094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.758039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:49.931 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-23T02:39:49.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T02:39:50.207 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:50.207 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":14,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T02:39:49.225833+0000","last_up_change":"2023-09-23T02:39:41.587263+0000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T02:39:43.121371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T02:39:47.647096+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6801","nonce":474982742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6803","nonce":474982742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6807","nonce":474982742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6805","nonce":474982742}]},"public_addr":"172.21.15.2:6801/474982742","cluster_addr":"172.21.15.2:6803/474982742","heartbeat_back_addr":"172.21.15.2:6807/474982742","heartbeat_front_addr":"172.21.15.2:6805/474982742","state":["exists","up"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6809","nonce":2376239628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6811","nonce":2376239628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6815","nonce":2376239628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6813","nonce":2376239628}]},"public_addr":"172.21.15.2:6809/2376239628","cluster_addr":"172.21.15.2:6811/2376239628","heartbeat_back_addr":"172.21.15.2:6815/2376239628","heartbeat_front_addr":"172.21.15.2:6813/2376239628","state":["exists","up"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6817","nonce":363956679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6819","nonce":363956679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6823","nonce":363956679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6821","nonce":363956679}]},"public_addr":"172.21.15.2:6817/363956679","cluster_addr":"172.21.15.2:6819/363956679","heartbeat_back_addr":"172.21.15.2:6823/363956679","heartbeat_front_addr":"172.21.15.2:6821/363956679","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.767365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.221094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.758039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T02:39:50.224 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-23T02:39:50.225 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-23T02:39:50.225 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-23T02:39:50.370 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2023-09-23T02:39:50.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-23T02:39:50.394 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2023-09-23T02:39:50.394 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-23T02:39:50.395 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2023-09-23T02:39:50.396 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-23T02:39:50.651 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2023-09-23T02:39:50.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-09-23T02:39:50.705 INFO:teuthology.orchestra.run.smithi002.stdout:34359738371 2023-09-23T02:39:50.709 INFO:teuthology.orchestra.run.smithi002.stdout:34359738371 2023-09-23T02:39:50.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-09-23T02:39:50.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-09-23T02:39:51.669 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-23T02:39:51.725 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-23T02:39:51.728 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-23T02:39:51.974 INFO:teuthology.orchestra.run.smithi002.stdout:34359738373 2023-09-23T02:39:51.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-09-23T02:39:51.992 DEBUG:teuthology.parallel:result is None 2023-09-23T02:39:52.008 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2023-09-23T02:39:52.013 INFO:teuthology.orchestra.run.smithi002.stdout:34359738372 2023-09-23T02:39:52.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-09-23T02:39:52.027 DEBUG:teuthology.parallel:result is None 2023-09-23T02:39:52.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-09-23T02:39:52.032 DEBUG:teuthology.parallel:result is None 2023-09-23T02:39:52.032 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-23T02:39:52.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T02:39:52.350 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:52.350 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T02:39:52.366 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-09-23T02:39:51.046483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95256,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020264,"statfs":{"total":289910292480,"available":289812750336,"internally_reserved":0,"allocated":1556480,"data_stored":1281896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":95743315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.825623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249548+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T02:39:49.249548+0000","last_peered":"2023-09-23T02:39:49.249548+0000","last_clean":"2023-09-23T02:39:49.249548+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T02:39:49.249548+0000","last_undegraded":"2023-09-23T02:39:49.249548+0000","last_fullsized":"2023-09-23T02:39:49.249548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:53:02.004199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249719+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T02:39:49.249719+0000","last_peered":"2023-09-23T02:39:49.249719+0000","last_clean":"2023-09-23T02:39:49.249719+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T02:39:49.249719+0000","last_undegraded":"2023-09-23T02:39:49.249719+0000","last_fullsized":"2023-09-23T02:39:49.249719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:47:05.320637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.248847+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T02:39:49.248847+0000","last_peered":"2023-09-23T02:39:49.248847+0000","last_clean":"2023-09-23T02:39:49.248847+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T02:39:49.248847+0000","last_undegraded":"2023-09-23T02:39:49.248847+0000","last_fullsized":"2023-09-23T02:39:49.248847+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:46:26.558917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249706+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T02:39:49.249706+0000","last_peered":"2023-09-23T02:39:49.249706+0000","last_clean":"2023-09-23T02:39:49.249706+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T02:39:49.249706+0000","last_undegraded":"2023-09-23T02:39:49.249706+0000","last_fullsized":"2023-09-23T02:39:49.249706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:07:15.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T02:39:50.234654+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T02:39:50.234654+0000","last_peered":"2023-09-23T02:39:50.234654+0000","last_clean":"2023-09-23T02:39:50.234654+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T02:39:50.234654+0000","last_undegraded":"2023-09-23T02:39:50.234654+0000","last_fullsized":"2023-09-23T02:39:50.234654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.250345+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T02:39:49.250345+0000","last_peered":"2023-09-23T02:39:49.250345+0000","last_clean":"2023-09-23T02:39:49.250345+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T02:39:49.250345+0000","last_undegraded":"2023-09-23T02:39:49.250345+0000","last_fullsized":"2023-09-23T02:39:49.250345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:51:42.444040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249825+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T02:39:49.249825+0000","last_peered":"2023-09-23T02:39:49.249825+0000","last_clean":"2023-09-23T02:39:49.249825+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T02:39:49.249825+0000","last_undegraded":"2023-09-23T02:39:49.249825+0000","last_fullsized":"2023-09-23T02:39:49.249825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:09:18.339355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249579+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T02:39:49.249579+0000","last_peered":"2023-09-23T02:39:49.249579+0000","last_clean":"2023-09-23T02:39:49.249579+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T02:39:49.249579+0000","last_undegraded":"2023-09-23T02:39:49.249579+0000","last_fullsized":"2023-09-23T02:39:49.249579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:20:09.693818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-23T02:39:48.223839+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T02:39:48.223839+0000","last_peered":"2023-09-23T02:39:48.223839+0000","last_clean":"2023-09-23T02:39:48.223839+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T02:39:48.223839+0000","last_undegraded":"2023-09-23T02:39:48.223839+0000","last_fullsized":"2023-09-23T02:39:48.223839+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T02:39:52.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T02:39:52.634 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:52.634 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T02:39:52.651 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-09-23T02:39:51.046483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95256,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020264,"statfs":{"total":289910292480,"available":289812750336,"internally_reserved":0,"allocated":1556480,"data_stored":1281896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":95743315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.825623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249548+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T02:39:49.249548+0000","last_peered":"2023-09-23T02:39:49.249548+0000","last_clean":"2023-09-23T02:39:49.249548+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T02:39:49.249548+0000","last_undegraded":"2023-09-23T02:39:49.249548+0000","last_fullsized":"2023-09-23T02:39:49.249548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:53:02.004199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249719+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T02:39:49.249719+0000","last_peered":"2023-09-23T02:39:49.249719+0000","last_clean":"2023-09-23T02:39:49.249719+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T02:39:49.249719+0000","last_undegraded":"2023-09-23T02:39:49.249719+0000","last_fullsized":"2023-09-23T02:39:49.249719+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:47:05.320637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.248847+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T02:39:49.248847+0000","last_peered":"2023-09-23T02:39:49.248847+0000","last_clean":"2023-09-23T02:39:49.248847+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T02:39:49.248847+0000","last_undegraded":"2023-09-23T02:39:49.248847+0000","last_fullsized":"2023-09-23T02:39:49.248847+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:46:26.558917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249706+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T02:39:49.249706+0000","last_peered":"2023-09-23T02:39:49.249706+0000","last_clean":"2023-09-23T02:39:49.249706+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T02:39:49.249706+0000","last_undegraded":"2023-09-23T02:39:49.249706+0000","last_fullsized":"2023-09-23T02:39:49.249706+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:07:15.737166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-23T02:39:50.234654+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T02:39:50.234654+0000","last_peered":"2023-09-23T02:39:50.234654+0000","last_clean":"2023-09-23T02:39:50.234654+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T02:39:50.234654+0000","last_undegraded":"2023-09-23T02:39:50.234654+0000","last_fullsized":"2023-09-23T02:39:50.234654+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.250345+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T02:39:49.250345+0000","last_peered":"2023-09-23T02:39:49.250345+0000","last_clean":"2023-09-23T02:39:49.250345+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T02:39:49.250345+0000","last_undegraded":"2023-09-23T02:39:49.250345+0000","last_fullsized":"2023-09-23T02:39:49.250345+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:51:42.444040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249825+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T02:39:49.249825+0000","last_peered":"2023-09-23T02:39:49.249825+0000","last_clean":"2023-09-23T02:39:49.249825+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T02:39:49.249825+0000","last_undegraded":"2023-09-23T02:39:49.249825+0000","last_fullsized":"2023-09-23T02:39:49.249825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:09:18.339355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-23T02:39:49.249579+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T02:39:49.249579+0000","last_peered":"2023-09-23T02:39:49.249579+0000","last_clean":"2023-09-23T02:39:49.249579+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T02:39:49.249579+0000","last_undegraded":"2023-09-23T02:39:49.249579+0000","last_fullsized":"2023-09-23T02:39:49.249579+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T08:20:09.693818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-09-23T02:39:48.223839+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T02:39:48.223839+0000","last_peered":"2023-09-23T02:39:48.223839+0000","last_clean":"2023-09-23T02:39:48.223839+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T02:39:48.223839+0000","last_undegraded":"2023-09-23T02:39:48.223839+0000","last_fullsized":"2023-09-23T02:39:48.223839+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32076,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339764,"statfs":{"total":96636764160,"available":96603918336,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":624088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T02:39:52.652 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-23T02:39:52.652 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-23T02:39:52.652 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-23T02:39:52.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-23T02:39:52.966 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T02:39:52.966 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-23T02:39:52.983 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-09-23T02:39:52.983 INFO:teuthology.run_tasks:Running task exec... 2023-09-23T02:39:52.993 INFO:teuthology.task.exec:Executing custom commands... 2023-09-23T02:39:52.993 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T02:39:52.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2' 2023-09-23T02:39:54.276 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4 4 erasure teuthologyprofile' 2023-09-23T02:39:56.267 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'datapool' created 2023-09-23T02:39:56.294 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set datapool allow_ec_overwrites true' 2023-09-23T02:39:57.273 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 3 allow_ec_overwrites to true 2023-09-23T02:39:57.302 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-09-23T02:40:00.328 INFO:teuthology.run_tasks:Running task install... 2023-09-23T02:40:00.336 DEBUG:teuthology.task.install:project ceph 2023-09-23T02:40:00.337 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'}} 2023-09-23T02:40:00.337 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'} 2023-09-23T02:40:00.337 INFO:teuthology.task.install:Using flavor: default 2023-09-23T02:40:00.342 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-23T02:40:00.342 INFO:teuthology.task.install:extra packages: [] 2023-09-23T02:40:00.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-key list | grep Ceph 2023-09-23T02:40:00.405 INFO:teuthology.orchestra.run.smithi002.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-09-23T02:40:00.435 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-23T02:40:00.435 INFO:teuthology.orchestra.run.smithi002.stdout:uid [ unknown] Ceph.com (release key) 2023-09-23T02:40:00.436 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-23T02:40:00.436 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-09-23T02:40:00.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:40:00.603 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default/ 2023-09-23T02:40:00.603 INFO:teuthology.task.install.deb:Package version is 18.0.0-6334-gee696c23-1jammy 2023-09-23T02:40:00.731 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:40:00.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-23T02:40:00.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo apt-get update 2023-09-23T02:40:00.949 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-09-23T02:40:00.960 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-09-23T02:40:00.980 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-09-23T02:40:01.037 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy InRelease 2023-09-23T02:40:01.064 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-09-23T02:40:01.073 INFO:teuthology.orchestra.run.smithi002.stdout:Hit:6 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy Release 2023-09-23T02:40:01.110 INFO:teuthology.orchestra.run.smithi002.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-62891/ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/ubuntu/jammy/flavors/default jammy Release.gpg 2023-09-23T02:40:02.213 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:40:02.227 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6334-gee696c23-1jammy cephadm=18.0.0-6334-gee696c23-1jammy ceph-mds=18.0.0-6334-gee696c23-1jammy ceph-mgr=18.0.0-6334-gee696c23-1jammy ceph-common=18.0.0-6334-gee696c23-1jammy ceph-fuse=18.0.0-6334-gee696c23-1jammy ceph-test=18.0.0-6334-gee696c23-1jammy ceph-volume=18.0.0-6334-gee696c23-1jammy radosgw=18.0.0-6334-gee696c23-1jammy python3-rados=18.0.0-6334-gee696c23-1jammy python3-rgw=18.0.0-6334-gee696c23-1jammy python3-cephfs=18.0.0-6334-gee696c23-1jammy python3-rbd=18.0.0-6334-gee696c23-1jammy libcephfs2=18.0.0-6334-gee696c23-1jammy libcephfs-dev=18.0.0-6334-gee696c23-1jammy librados2=18.0.0-6334-gee696c23-1jammy librbd1=18.0.0-6334-gee696c23-1jammy rbd-fuse=18.0.0-6334-gee696c23-1jammy 2023-09-23T02:40:02.289 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:40:02.478 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-09-23T02:40:02.479 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-09-23T02:40:02.648 INFO:teuthology.orchestra.run.smithi002.stdout:ceph is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-common is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-fuse is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-mds is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-mgr is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-test is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-volume is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.649 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:libcephfs-dev is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:libcephfs2 is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:librados2 is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:librbd1 is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:python3-cephfs is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:python3-rados is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:python3-rbd is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.650 INFO:teuthology.orchestra.run.smithi002.stdout:python3-rgw is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout:radosgw is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout:rbd-fuse is already the newest version (18.0.0-6334-gee696c23-1jammy). 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-09-23T02:40:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-09-23T02:40:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-09-23T02:40:02.652 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-23T02:40:02.689 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 0 newly installed, 0 to remove and 253 not upgraded. 2023-09-23T02:40:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-23T02:40:02.691 DEBUG:teuthology.orchestra.run.smithi002:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-09-23T02:40:02.753 INFO:teuthology.orchestra.run.smithi002.stdout:Reading package lists... 2023-09-23T02:40:02.942 INFO:teuthology.orchestra.run.smithi002.stdout:Building dependency tree... 2023-09-23T02:40:02.942 INFO:teuthology.orchestra.run.smithi002.stdout:Reading state information... 2023-09-23T02:40:03.110 INFO:teuthology.orchestra.run.smithi002.stdout:The following packages were automatically installed and are no longer required: 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: python2.7 python2.7-dev python2.7-minimal 2023-09-23T02:40:03.111 INFO:teuthology.orchestra.run.smithi002.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-23T02:40:03.112 INFO:teuthology.orchestra.run.smithi002.stdout:The following additional packages will be installed: 2023-09-23T02:40:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemblk1 2023-09-23T02:40:03.113 INFO:teuthology.orchestra.run.smithi002.stdout:Suggested packages: 2023-09-23T02:40:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: gnuplot gfio python-scipy 2023-09-23T02:40:03.137 INFO:teuthology.orchestra.run.smithi002.stdout:The following NEW packages will be installed: 2023-09-23T02:40:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: fio libpmemblk1 2023-09-23T02:40:03.195 INFO:teuthology.orchestra.run.smithi002.stdout:0 upgraded, 2 newly installed, 0 to remove and 253 not upgraded. 2023-09-23T02:40:03.195 INFO:teuthology.orchestra.run.smithi002.stdout:Need to get 4,112 kB of archives. 2023-09-23T02:40:03.195 INFO:teuthology.orchestra.run.smithi002.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-09-23T02:40:03.195 INFO:teuthology.orchestra.run.smithi002.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-09-23T02:40:03.262 INFO:teuthology.orchestra.run.smithi002.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-09-23T02:40:03.784 INFO:teuthology.orchestra.run.smithi002.stdout:Fetched 4,112 kB in 0s (15.0 MB/s) 2023-09-23T02:40:03.952 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-09-23T02:40:04.057 INFO:teuthology.orchestra.run.smithi002.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172896 files and directories currently installed.) 2023-09-23T02:40:04.063 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-09-23T02:40:04.142 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-09-23T02:40:04.722 INFO:teuthology.orchestra.run.smithi002.stdout:Selecting previously unselected package fio. 2023-09-23T02:40:04.749 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-09-23T02:40:04.779 INFO:teuthology.orchestra.run.smithi002.stdout:Unpacking fio (3.28-1) ... 2023-09-23T02:40:05.307 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-09-23T02:40:05.443 INFO:teuthology.orchestra.run.smithi002.stdout:Setting up fio (3.28-1) ... 2023-09-23T02:40:05.988 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-09-23T02:40:06.946 INFO:teuthology.orchestra.run.smithi002.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-09-23T02:40:07.421 INFO:teuthology.orchestra.run.smithi002.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-23T02:40:07.424 DEBUG:teuthology.parallel:result is None 2023-09-23T02:40:07.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:40:07.568 DEBUG:teuthology.orchestra.run.smithi002:> dpkg-query -W -f '${Version}' ceph 2023-09-23T02:40:07.589 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-6334-gee696c23-1jammy 2023-09-23T02:40:07.590 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6334-gee696c23-1jammy 2023-09-23T02:40:07.590 INFO:teuthology.task.install:The correct ceph version 18.0.0-6334-gee696c23-1jammy is installed. 2023-09-23T02:40:07.590 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-09-23T02:40:07.590 INFO:teuthology.run_tasks:Running task workunit... 2023-09-23T02:40:07.605 INFO:tasks.workunit:Pulling workunits from ref ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:40:07.606 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-23T02:40:07.606 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-23T02:40:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T02:40:07.637 INFO:teuthology.orchestra.run.smithi002.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-23T02:40:07.638 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-23T02:40:07.684 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-23T02:40:07.685 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-23T02:40:07.733 INFO:tasks.workunit:timeout=3h 2023-09-23T02:40:07.733 INFO:tasks.workunit:cleanup=True 2023-09-23T02:40:07.734 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e 2023-09-23T02:40:07.782 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-23T02:41:20.917 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 99% (11937/11967) Updating files: 100% (11967/11967) Updating files: 100% (11967/11967), done. 2023-09-23T02:41:21.672 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to 'ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e'. 2023-09-23T02:41:21.672 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2023-09-23T02:41:21.673 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.674 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2023-09-23T02:41:21.674 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.674 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-23T02:41:21.674 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-09-23T02:41:21.674 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at ee696c23a60 qa/suites/rbd: add test to check rbd_support module recovery 2023-09-23T02:41:21.677 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-23T02:41:21.726 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-23T02:41:21.729 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-23T02:41:21.729 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-23T02:41:22.528 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-23T02:41:22.604 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-23T02:41:22.665 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-23T02:41:22.667 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-23T02:41:22.667 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-23T02:41:22.717 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-23T02:41:22.720 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-09-23T02:41:22.721 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-23T02:41:22.770 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-09-23T02:41:22.771 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-09-23T02:41:22.771 DEBUG:teuthology.orchestra.run.smithi002:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-09-23T02:41:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ POOL=rbd 2023-09-23T02:41:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ IMAGE_PREFIX=image 2023-09-23T02:41:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ MNT_DIR_PREFIX=mnt 2023-09-23T02:41:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ NUM_IMAGES=20 2023-09-23T02:41:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ RUN_TIME=3600 2023-09-23T02:41:22.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror pool enable rbd image 2023-09-23T02:41:22.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror pool peer add rbd dummy 2023-09-23T02:41:22.910 INFO:tasks.workunit.client.0.smithi002.stdout:c46a23df-887c-417a-adb3-1a28b68df068 2023-09-23T02:41:22.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i = 1 )) 2023-09-23T02:41:22.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:22.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image1 2023-09-23T02:41:22.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-09-23T02:41:23.351 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:23.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-09-23T02:41:23.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:23.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:23.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image2 2023-09-23T02:41:23.484 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-09-23T02:41:24.353 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:24.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-09-23T02:41:24.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:24.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:24.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image3 2023-09-23T02:41:24.484 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-09-23T02:41:25.358 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:25.376 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-09-23T02:41:25.437 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:25.438 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:25.438 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image4 2023-09-23T02:41:25.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-09-23T02:41:26.374 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:26.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-09-23T02:41:26.450 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:26.450 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:26.450 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image5 2023-09-23T02:41:26.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-09-23T02:41:27.376 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:27.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-09-23T02:41:27.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:27.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:27.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image6 2023-09-23T02:41:27.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-09-23T02:41:28.388 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:28.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-09-23T02:41:28.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:28.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:28.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image7 2023-09-23T02:41:28.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-09-23T02:41:29.399 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:29.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-09-23T02:41:29.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:29.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:29.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image8 2023-09-23T02:41:29.529 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-09-23T02:41:30.406 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:30.423 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-09-23T02:41:30.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:30.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:30.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image9 2023-09-23T02:41:30.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-09-23T02:41:31.417 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:31.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-09-23T02:41:31.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:31.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:31.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image10 2023-09-23T02:41:31.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-09-23T02:41:32.414 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:32.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-09-23T02:41:32.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:32.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:32.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image11 2023-09-23T02:41:32.545 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-09-23T02:41:33.419 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:33.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-09-23T02:41:33.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:33.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:33.498 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image12 2023-09-23T02:41:33.545 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-09-23T02:41:34.430 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:34.446 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-09-23T02:41:34.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:34.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:34.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image13 2023-09-23T02:41:34.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-09-23T02:41:35.445 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:35.460 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-09-23T02:41:35.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:35.523 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:35.523 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image14 2023-09-23T02:41:35.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-09-23T02:41:36.449 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:36.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-09-23T02:41:36.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:36.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:36.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image15 2023-09-23T02:41:36.577 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-09-23T02:41:37.458 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:37.474 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-09-23T02:41:37.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:37.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:37.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image16 2023-09-23T02:41:37.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-09-23T02:41:38.471 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:38.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-09-23T02:41:38.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:38.555 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:38.555 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image17 2023-09-23T02:41:38.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-09-23T02:41:39.465 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:39.482 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-09-23T02:41:39.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:39.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:39.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image18 2023-09-23T02:41:39.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-09-23T02:41:40.477 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:40.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-09-23T02:41:40.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:40.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:40.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image19 2023-09-23T02:41:40.607 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-09-23T02:41:41.491 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:41.507 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-09-23T02:41:41.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:41.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:41.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd create --size 1G rbd/image20 2023-09-23T02:41:41.619 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-09-23T02:41:42.492 INFO:tasks.workunit.client.0.smithi002.stdout:Mirroring enabled 2023-09-23T02:41:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-09-23T02:41:42.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i = 1 )) 2023-09-23T02:41:42.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.573 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image1 2023-09-23T02:41:42.686 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd0 2023-09-23T02:41:42.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:42.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image2 2023-09-23T02:41:42.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd1 2023-09-23T02:41:42.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:42.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image3 2023-09-23T02:41:42.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd2 2023-09-23T02:41:42.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:42.834 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image4 2023-09-23T02:41:42.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd3 2023-09-23T02:41:42.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:42.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image5 2023-09-23T02:41:42.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd4 2023-09-23T02:41:42.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:42.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:42.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:42.983 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image6 2023-09-23T02:41:43.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd5 2023-09-23T02:41:43.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:43.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:43.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:43.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image7 2023-09-23T02:41:43.309 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd6 2023-09-23T02:41:43.309 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:43.309 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:43.309 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:43.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image8 2023-09-23T02:41:43.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd7 2023-09-23T02:41:43.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:43.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:43.497 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image9 2023-09-23T02:41:43.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:43.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd8 2023-09-23T02:41:43.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:43.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:43.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image10 2023-09-23T02:41:43.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:43.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd9 2023-09-23T02:41:43.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:43.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:43.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:43.937 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image11 2023-09-23T02:41:44.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd10 2023-09-23T02:41:44.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:44.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:44.164 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:44.164 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image12 2023-09-23T02:41:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd11 2023-09-23T02:41:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image13 2023-09-23T02:41:44.518 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd12 2023-09-23T02:41:44.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:44.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:44.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:44.520 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image14 2023-09-23T02:41:44.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd13 2023-09-23T02:41:44.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:44.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:44.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image15 2023-09-23T02:41:44.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:45.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd14 2023-09-23T02:41:45.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:45.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:45.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image16 2023-09-23T02:41:45.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:45.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd15 2023-09-23T02:41:45.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:45.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:45.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:45.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image17 2023-09-23T02:41:45.530 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd16 2023-09-23T02:41:45.531 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:45.531 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:45.531 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:45.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image18 2023-09-23T02:41:45.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd17 2023-09-23T02:41:45.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:45.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:45.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:45.797 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image19 2023-09-23T02:41:46.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd18 2023-09-23T02:41:46.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:46.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:46.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:46.034 INFO:tasks.workunit.client.0.smithi002.stderr:++ sudo rbd device map rbd/image20 2023-09-23T02:41:46.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ DEVS[$i]=/dev/rbd19 2023-09-23T02:41:46.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T02:41:46.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 20 )) 2023-09-23T02:41:46.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --size=800M --iodepth=2 --runtime=43200 --time_based 2023-09-23T02:41:46.293 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:41:46.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436906 2023-09-23T02:41:46.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ END_TIME=1695440506 2023-09-23T02:41:46.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT= 2023-09-23T02:41:46.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR= 2023-09-23T02:41:46.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436906 -le 1695440506 ]] 2023-09-23T02:41:46.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z '' ]] 2023-09-23T02:41:46.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:41:54.289 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:41:54.286+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8a2aa5b0 2023-09-23T02:41:54.290 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:41:54.286+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8aa28270 2023-09-23T02:41:54.290 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:41:54.286+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8afa6750 2023-09-23T02:41:54.290 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:41:54.286+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8c639d40 2023-09-23T02:41:56.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:41:56.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:41:56.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:41:56.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:41:57.475 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/726649277 2023-09-23T02:41:57.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:41:57.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436917 2023-09-23T02:41:57.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436917 -le 1695440506 ]] 2023-09-23T02:41:57.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/726649277 ]] 2023-09-23T02:41:57.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/726649277 != '' ]] 2023-09-23T02:41:57.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/726649277 2023-09-23T02:41:59.278 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/726649277 until 2023-09-23T03:41:58.526165+0000 (3600 sec) 2023-09-23T02:41:59.313 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:41:59.316 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/726649277 2023-09-23T02:42:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.002+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.002+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aeee200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.006+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.006+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aeeff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.018+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.018+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16addcb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.026+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.026+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5db000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.034+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.034+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5da280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.054+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.054+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5db900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.078+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.078+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5daf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.086+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.086+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5da480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.094+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.094+0000 7f3994a27640 -1 librbd::ImageState: 0x55c163a94280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.110 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.106+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.110 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.106+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16addd480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.114+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.114+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aeef780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.122+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.122+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5da480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.141 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.138+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.141 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.138+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5db580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.152 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.150+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.152 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:42:00.150+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5db900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:42:00.217 INFO:tasks.workunit.client.0.smithi002.stderr:listed 1 entries 2023-09-23T02:42:00.280 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/726649277 2023-09-23T02:42:00.280 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:42:10.294 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:42:10.299 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:42:10.302 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:42:10.302 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:42:11.545 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/726649277 2023-09-23T02:42:11.545 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:42:11.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436931 2023-09-23T02:42:11.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436931 -le 1695440506 ]] 2023-09-23T02:42:11.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/726649277 ]] 2023-09-23T02:42:11.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/726649277 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\2\6\6\4\9\2\7\7 ]] 2023-09-23T02:42:11.547 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:42:21.549 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:42:21.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:42:21.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:42:21.570 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:42:22.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/726649277 2023-09-23T02:42:22.969 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:42:22.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436942 2023-09-23T02:42:22.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436942 -le 1695440506 ]] 2023-09-23T02:42:22.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/726649277 ]] 2023-09-23T02:42:22.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/726649277 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\2\6\6\4\9\2\7\7 ]] 2023-09-23T02:42:22.971 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:42:32.973 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:42:32.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:42:32.979 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:42:32.987 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:42:34.279 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/726649277 2023-09-23T02:42:34.285 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:42:34.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436954 2023-09-23T02:42:34.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436954 -le 1695440506 ]] 2023-09-23T02:42:34.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/726649277 ]] 2023-09-23T02:42:34.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/726649277 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\2\6\6\4\9\2\7\7 ]] 2023-09-23T02:42:34.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:42:44.291 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:42:44.292 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:42:44.301 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:42:44.326 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:42:45.628 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/726649277 2023-09-23T02:42:45.629 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:42:45.630 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436965 2023-09-23T02:42:45.630 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436965 -le 1695440506 ]] 2023-09-23T02:42:45.630 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/726649277 ]] 2023-09-23T02:42:45.630 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/726649277 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\2\6\6\4\9\2\7\7 ]] 2023-09-23T02:42:45.630 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:42:55.631 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:42:55.632 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:42:55.632 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:42:55.654 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:42:56.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:42:58.420 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:42:58.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436976 2023-09-23T02:42:58.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436976 -le 1695440506 ]] 2023-09-23T02:42:58.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:42:58.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\2\6\6\4\9\2\7\7 ]] 2023-09-23T02:42:58.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/208262749 2023-09-23T02:42:59.358 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/208262749 until 2023-09-23T03:42:58.661083+0000 (3600 sec) 2023-09-23T02:42:59.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:42:59.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/208262749 2023-09-23T02:43:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.008+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.008+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b4c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.020+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.020+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b5db980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.028+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.028+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.034 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.028+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.034 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.028+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.032+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.032+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.036+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.036+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.048+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.048+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b4c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.056+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.056+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.070 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.064+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.070 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.064+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b5db980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.076+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.076+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.092+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.103 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.100+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.100+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.120 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.116+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.121 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.116+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b4e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.128+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.128+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.137 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.132+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.137 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.132+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.149 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.144+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.149 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:43:00.144+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b9b5b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:43:00.423 INFO:tasks.workunit.client.0.smithi002.stderr:listed 2 entries 2023-09-23T02:43:00.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/208262749 2023-09-23T02:43:00.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:43:10.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:43:10.480 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:43:10.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:43:10.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:43:11.580 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:43:11.581 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:43:11.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695436991 2023-09-23T02:43:11.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695436991 -le 1695440506 ]] 2023-09-23T02:43:11.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:43:11.583 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:43:11.583 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:43:12.367 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:43:12.360+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:43:12.367 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:43:12.360+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:43:21.588 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:43:21.589 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:43:21.600 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:43:21.602 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:43:22.840 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:43:22.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:43:22.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437002 2023-09-23T02:43:22.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437002 -le 1695440506 ]] 2023-09-23T02:43:22.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:43:22.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:43:22.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:43:32.847 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:43:32.847 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:43:32.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:43:32.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:43:34.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:43:34.230 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:43:34.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437014 2023-09-23T02:43:34.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437014 -le 1695440506 ]] 2023-09-23T02:43:34.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:43:34.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:43:34.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:43:44.240 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437025 2023-09-23T02:43:45.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437025 -le 1695440506 ]] 2023-09-23T02:43:45.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:43:45.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:43:45.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:43:55.216 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:43:55.218 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:43:55.219 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:43:55.227 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:43:56.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/208262749 2023-09-23T02:43:56.703 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:43:56.704 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437036 2023-09-23T02:43:56.705 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437036 -le 1695440506 ]] 2023-09-23T02:43:56.705 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/208262749 ]] 2023-09-23T02:43:56.705 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/208262749 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:43:56.705 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:44:01.592 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:44:01.593 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:44:01.594 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:01.591+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:44:04.832 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:44:04.831+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:44:06.713 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437047 2023-09-23T02:44:07.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437047 -le 1695440506 ]] 2023-09-23T02:44:07.725 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:44:07.725 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\8\2\6\2\7\4\9 ]] 2023-09-23T02:44:07.725 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2810758585 2023-09-23T02:44:09.378 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2810758585 until 2023-09-23T03:44:08.581996+0000 (3600 sec) 2023-09-23T02:44:09.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2810758585 2023-09-23T02:44:09.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:44:10.082 INFO:tasks.workunit.client.0.smithi002.stderr:listed 3 entries 2023-09-23T02:44:10.122 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2810758585 2023-09-23T02:44:10.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:44:12.228 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:44:12.227+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8afa6750 2023-09-23T02:44:12.228 INFO:tasks.ceph.osd.2.smithi002.stderr:2023-09-23T02:44:12.227+0000 7f3c4c4ef640 -1 reset not still connected to 0x561b8c639d40 2023-09-23T02:44:20.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:44:20.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:44:20.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:44:20.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:44:21.146 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437061 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437061 -le 1695440506 ]] 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:44:21.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:44:31.160 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:44:31.161 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:44:31.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:44:31.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:44:32.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:44:32.186 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:44:32.186 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437072 2023-09-23T02:44:32.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437072 -le 1695440506 ]] 2023-09-23T02:44:32.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:44:32.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:44:32.187 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:44:42.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:44:42.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:44:42.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:44:42.194 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:44:43.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:44:43.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:44:43.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437083 2023-09-23T02:44:43.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437083 -le 1695440506 ]] 2023-09-23T02:44:43.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:44:43.326 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:44:43.326 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:44:53.327 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:44:53.331 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:44:53.334 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:44:53.338 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:44:54.364 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:44:54.365 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:44:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437094 2023-09-23T02:44:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437094 -le 1695440506 ]] 2023-09-23T02:44:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:44:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:44:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:45:04.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:45:04.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:45:04.371 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:45:04.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:45:05.639 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2810758585 2023-09-23T02:45:05.639 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:45:05.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437105 2023-09-23T02:45:05.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437105 -le 1695440506 ]] 2023-09-23T02:45:05.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2810758585 ]] 2023-09-23T02:45:05.641 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2810758585 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:45:05.641 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:45:15.642 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:45:15.646 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:45:15.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:45:15.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:45:17.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:45:17.017 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:45:17.019 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437117 2023-09-23T02:45:17.019 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437117 -le 1695440506 ]] 2023-09-23T02:45:17.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:45:17.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\1\0\7\5\8\5\8\5 ]] 2023-09-23T02:45:17.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/703823873 2023-09-23T02:45:19.380 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/703823873 until 2023-09-23T03:45:18.726351+0000 (3600 sec) 2023-09-23T02:45:19.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:45:19.436 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/703823873 2023-09-23T02:45:20.566 INFO:tasks.workunit.client.0.smithi002.stderr:listed 4 entries 2023-09-23T02:45:20.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/703823873 2023-09-23T02:45:20.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:45:30.618 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:45:30.619 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:45:30.619 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:45:30.622 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:45:31.819 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:45:31.819 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:45:31.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437131 2023-09-23T02:45:31.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437131 -le 1695440506 ]] 2023-09-23T02:45:31.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:45:31.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:45:31.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:45:41.823 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:45:41.823 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:45:41.824 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:45:41.826 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:45:43.115 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:45:43.115 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:45:43.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437143 2023-09-23T02:45:43.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437143 -le 1695440506 ]] 2023-09-23T02:45:43.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:45:43.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:45:43.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:45:52.066 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:45:52.066 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:45:52.066 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:45:52.066 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:45:52.067 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:45:52.060+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:45:53.118 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:45:53.120 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:45:53.122 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:45:53.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:45:54.470 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:45:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:45:54.472 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437154 2023-09-23T02:45:54.472 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437154 -le 1695440506 ]] 2023-09-23T02:45:54.472 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:45:54.472 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:45:54.472 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:46:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.004+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.004+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.012+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.012+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.012+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.012+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.032+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.032+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.048+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.048+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.056+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.056+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.072+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.072+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.076+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.076+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.084 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.080+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.084 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.080+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.100+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.100+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.104+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.104+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.116 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.112+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.116 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.112+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.124+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.124+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.133 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.128+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.133 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.128+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.138 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.136+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.138 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.136+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.148 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.144+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.148 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.144+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.150 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.148+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:00.150 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:46:00.148+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:46:04.479 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:46:04.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:46:04.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:46:04.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:46:05.621 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:46:05.622 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:46:05.623 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437165 2023-09-23T02:46:05.623 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437165 -le 1695440506 ]] 2023-09-23T02:46:05.623 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:46:05.624 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:46:05.624 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:46:15.631 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:46:15.631 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:46:15.631 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:46:15.646 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:46:16.608 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/703823873 2023-09-23T02:46:16.608 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:46:16.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437176 2023-09-23T02:46:16.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437176 -le 1695440506 ]] 2023-09-23T02:46:16.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/703823873 ]] 2023-09-23T02:46:16.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/703823873 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:46:16.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:46:26.616 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:46:26.616 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:46:26.618 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:46:26.644 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:46:27.690 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:46:27.695 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:46:27.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437187 2023-09-23T02:46:27.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437187 -le 1695440506 ]] 2023-09-23T02:46:27.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:46:27.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\0\3\8\2\3\8\7\3 ]] 2023-09-23T02:46:27.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2718665785 2023-09-23T02:46:30.417 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2718665785 until 2023-09-23T03:46:29.748739+0000 (3600 sec) 2023-09-23T02:46:30.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2718665785 2023-09-23T02:46:30.507 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:46:31.614 INFO:tasks.workunit.client.0.smithi002.stderr:listed 5 entries 2023-09-23T02:46:31.650 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2718665785 2023-09-23T02:46:31.650 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:46:41.658 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:46:41.658 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:46:41.667 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:46:41.680 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:46:43.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:46:43.097 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:46:43.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437203 2023-09-23T02:46:43.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437203 -le 1695440506 ]] 2023-09-23T02:46:43.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:46:43.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:46:43.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:46:53.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:46:53.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:46:53.110 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:46:53.120 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:46:54.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:46:54.469 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:46:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437214 2023-09-23T02:46:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437214 -le 1695440506 ]] 2023-09-23T02:46:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:46:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:46:54.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:47:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.004+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.004+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.004+0000 7f3994226640 -1 librbd::ImageState: 0x55c16aeef080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.004+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb75f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.008+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.008+0000 7f3994226640 -1 librbd::ImageState: 0x55c16ba96500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.012+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.012+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb75c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.024+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.024+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.053 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.048+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.048+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.060+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.060+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.068+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.068+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.080+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.080+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.084+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.084+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.092+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.100+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.100+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.104+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.104+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.116+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.116+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.125 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.120+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.125 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.120+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.148 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.144+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.149 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.144+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.160+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.160+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.170 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.168+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.170 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.168+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.174 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.172+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:00.174 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:00.172+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:47:04.363 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:47:04.364 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:47:04.364 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:04.356+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:47:04.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:47:04.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:47:04.481 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:47:04.485 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:47:05.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:47:05.803 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:47:05.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437225 2023-09-23T02:47:05.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437225 -le 1695440506 ]] 2023-09-23T02:47:05.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:47:05.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:47:05.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:47:15.807 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:47:15.814 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:47:15.815 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:47:15.831 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:47:16.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:47:16.965 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:47:16.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437236 2023-09-23T02:47:16.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437236 -le 1695440506 ]] 2023-09-23T02:47:16.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:47:16.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:47:16.966 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:47:26.969 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:47:26.970 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:47:26.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:47:26.978 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:47:27.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2718665785 2023-09-23T02:47:27.959 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:47:27.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437247 2023-09-23T02:47:27.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437247 -le 1695440506 ]] 2023-09-23T02:47:27.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2718665785 ]] 2023-09-23T02:47:27.961 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2718665785 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:47:27.961 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:47:30.484 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.479+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:47:30.492 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:47:30.493 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:47:30.493 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:47:30.494 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:47:30.494 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:47:30.494 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:47:30.494 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:30.483+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:47:37.971 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:47:37.972 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:47:37.972 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:47:37.988 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:47:39.270 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:47:39.270 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:47:39.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437259 2023-09-23T02:47:39.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437259 -le 1695440506 ]] 2023-09-23T02:47:39.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:47:39.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\7\1\8\6\6\5\7\8\5 ]] 2023-09-23T02:47:39.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1409823025 2023-09-23T02:47:41.412 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1409823025 until 2023-09-23T03:47:40.798405+0000 (3600 sec) 2023-09-23T02:47:41.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:47:41.470 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1409823025 2023-09-23T02:47:42.692 INFO:tasks.workunit.client.0.smithi002.stderr:listed 6 entries 2023-09-23T02:47:42.737 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1409823025 2023-09-23T02:47:42.737 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:47:49.877 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.871+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:47:49.878 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.875+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:47:49.879 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.875+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:47:49.886 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:47:49.886 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:47:49.886 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:47:49.886 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:47:49.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:49.883+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:47:50.365 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:47:50.369 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:47:50.369 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:47:50.370 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:47:50.363+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:47:52.738 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:47:52.738 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:47:52.750 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:47:52.756 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:47:53.939 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:47:53.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:47:53.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437273 2023-09-23T02:47:53.940 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437273 -le 1695440506 ]] 2023-09-23T02:47:53.941 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:47:53.941 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:47:53.941 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:48:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:59.999+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:59.999+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:59.999+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:47:59.999+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf09c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.003+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.003+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf09e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.011+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.011+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.023+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.023+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.023+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.023+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.035+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.035+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.039+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.039+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.055+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.055+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaaa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.066 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.063+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.066 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.063+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.067+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.067+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.071+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.071+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbab680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.079+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.079+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbab880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.087+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.087+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.096 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.091+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.096 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.091+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbabc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.099+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.099+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbabe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.103+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.103+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.107+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.107+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.111+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.111+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.126 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.123+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:00.126 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:48:00.123+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:48:03.943 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:48:03.946 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:48:03.953 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:48:03.957 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:48:05.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:48:05.419 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:48:05.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437285 2023-09-23T02:48:05.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437285 -le 1695440506 ]] 2023-09-23T02:48:05.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:48:05.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:48:05.421 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:48:15.425 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:48:15.427 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:48:15.432 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:48:15.443 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:48:16.663 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:48:16.668 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:48:16.669 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437296 2023-09-23T02:48:16.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437296 -le 1695440506 ]] 2023-09-23T02:48:16.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:48:16.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:48:16.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:48:26.671 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:48:26.672 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:48:26.672 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:48:26.712 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:48:27.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:48:27.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:48:27.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437307 2023-09-23T02:48:27.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437307 -le 1695440506 ]] 2023-09-23T02:48:27.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:48:27.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:48:27.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:48:37.997 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:48:38.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:48:38.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:48:38.026 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:48:39.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1409823025 2023-09-23T02:48:39.340 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:48:39.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437319 2023-09-23T02:48:39.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437319 -le 1695440506 ]] 2023-09-23T02:48:39.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1409823025 ]] 2023-09-23T02:48:39.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1409823025 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:48:39.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:48:49.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:48:49.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:48:49.349 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:48:49.374 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:48:50.666 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:48:50.666 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:48:50.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437330 2023-09-23T02:48:50.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437330 -le 1695440506 ]] 2023-09-23T02:48:50.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:48:50.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\4\0\9\8\2\3\0\2\5 ]] 2023-09-23T02:48:50.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2884784619 2023-09-23T02:48:53.432 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2884784619 until 2023-09-23T03:48:52.854134+0000 (3600 sec) 2023-09-23T02:48:53.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2884784619 2023-09-23T02:48:53.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:48:54.578 INFO:tasks.workunit.client.0.smithi002.stderr:listed 7 entries 2023-09-23T02:48:54.615 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2884784619 2023-09-23T02:48:54.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:49:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.003+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.003+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbab100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.007+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.007+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbaaf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.011+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.011+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16be93200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.015+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.015+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbcbd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.027+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.027+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16be92480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.039+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.039+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aef3e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.043+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.043+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf08700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.047+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.047+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16be93100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.055+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.055+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf09780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.063+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.063+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf09900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.079+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.079+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf09680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.084 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.087+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.087+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bad6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.087+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.087+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bb74900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.091+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.090 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.091+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf08780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.095+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.095+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf08e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.099+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.099+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef1080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.103+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.103 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.103+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef0e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.107 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.107+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.107 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.107+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef0c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.115+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.115+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aeee780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.115+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:49:00.115+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef1e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:49:04.617 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:49:04.617 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:49:04.617 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:49:04.637 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:49:05.690 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:49:05.696 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:49:05.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437345 2023-09-23T02:49:05.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437345 -le 1695440506 ]] 2023-09-23T02:49:05.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:49:05.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:49:05.697 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:49:15.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:49:15.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:49:15.702 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:49:15.731 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:49:17.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:49:17.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:49:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437357 2023-09-23T02:49:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437357 -le 1695440506 ]] 2023-09-23T02:49:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:49:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:49:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:49:27.024 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:49:27.024 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:49:27.024 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:49:27.024 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:49:28.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:49:28.255 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:49:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437368 2023-09-23T02:49:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437368 -le 1695440506 ]] 2023-09-23T02:49:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:49:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:49:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:49:38.258 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:49:38.259 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:49:38.263 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:49:38.273 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:49:39.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:49:39.576 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:49:39.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437379 2023-09-23T02:49:39.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437379 -le 1695440506 ]] 2023-09-23T02:49:39.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:49:39.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:49:39.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:49:49.588 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:49:49.597 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:49:49.601 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:49:49.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:49:50.672 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2884784619 2023-09-23T02:49:50.673 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:49:50.673 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437390 2023-09-23T02:49:50.673 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437390 -le 1695440506 ]] 2023-09-23T02:49:50.673 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2884784619 ]] 2023-09-23T02:49:50.674 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2884784619 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:49:50.674 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:50:00.675 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:50:00.675 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:50:00.675 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:50:00.676 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:50:01.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:50:01.988 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:50:01.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437401 2023-09-23T02:50:01.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437401 -le 1695440506 ]] 2023-09-23T02:50:01.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:50:01.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\8\4\7\8\4\6\1\9 ]] 2023-09-23T02:50:01.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2953468242 2023-09-23T02:50:03.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16b60f200 image watch failed: 94289231307776, (107) Transport endpoint is not connected 2023-09-23T02:50:03.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3994226640 -1 librbd::Watcher: 0x55c16b60f200 handle_error: handle=94289231307776: (107) Transport endpoint is not connected 2023-09-23T02:50:03.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c128600 image watch failed: 94289230185472, (107) Transport endpoint is not connected 2023-09-23T02:50:03.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c128600 handle_error: handle=94289230185472: (107) Transport endpoint is not connected 2023-09-23T02:50:03.472 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16bab8230 handle_unwatch client blocklisted 2023-09-23T02:50:03.472 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16b60f200 handle_rewatch: client blocklisted 2023-09-23T02:50:03.472 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16aee4820 handle_unwatch client blocklisted 2023-09-23T02:50:03.472 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c128600 handle_rewatch: client blocklisted 2023-09-23T02:50:03.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16b60f500 image watch failed: 94289229072384, (107) Transport endpoint is not connected 2023-09-23T02:50:03.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16b60f500 handle_error: handle=94289229072384: (107) Transport endpoint is not connected 2023-09-23T02:50:03.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16aee5900 handle_unwatch client blocklisted 2023-09-23T02:50:03.474 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.474+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16b60f500 handle_rewatch: client blocklisted 2023-09-23T02:50:03.475 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.478+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c128000 image watch failed: 94289231306752, (107) Transport endpoint is not connected 2023-09-23T02:50:03.475 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.478+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c128000 handle_error: handle=94289231306752: (107) Transport endpoint is not connected 2023-09-23T02:50:03.479 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.482+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16afad590 handle_unwatch client blocklisted 2023-09-23T02:50:03.479 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:03.482+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c128000 handle_rewatch: client blocklisted 2023-09-23T02:50:04.463 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2953468242 until 2023-09-23T03:50:03.686628+0000 (3600 sec) 2023-09-23T02:50:04.469 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3993a25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.469 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a2410 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.470 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c22db80 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.470 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.470 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.470 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a2b60 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c23ba20 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a2a90 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.471 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.470+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c1afe40 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.472 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c00e6e0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16bceb240 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c204420 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c205e40 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c03eec0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3994226640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c123c00 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.473 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.474 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.474 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bfa5da0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.474 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bfa6780 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.474 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:04.474+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16b6ec120 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:04.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:50:04.514 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2953468242 2023-09-23T02:50:05.291 INFO:tasks.workunit.client.0.smithi002.stderr:listed 8 entries 2023-09-23T02:50:05.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2953468242 2023-09-23T02:50:05.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:50:13.340 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::image_watcher::NotifyLockOwner: 0x55c16bb75080 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:13.939 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16b60f500 error requesting lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:13.939 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16b60f500 blocklisted waiting for exclusive lock 2023-09-23T02:50:13.939 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::ManagedLock: 0x55c16b6804f8 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T02:50:13.939 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3040 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:13.939 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:50:13.342+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c22d1e0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.878+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:50:13.941 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.882+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:50:13.942 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.890+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:50:13.942 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.890+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:50:13.942 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.894+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:50:13.942 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.894+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:50:13.942 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:13.894+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:50:15.330 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:50:15.330 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:50:15.330 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:50:15.358 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:50:16.412 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:50:16.413 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:50:16.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437416 2023-09-23T02:50:16.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437416 -le 1695440506 ]] 2023-09-23T02:50:16.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:50:16.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:50:16.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:50:17.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:17.042+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:50:17.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:17.042+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:50:17.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:17.042+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:50:26.419 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:50:26.419 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:50:26.425 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:50:26.431 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:50:27.537 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:50:27.543 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:50:27.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437427 2023-09-23T02:50:27.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437427 -le 1695440506 ]] 2023-09-23T02:50:27.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:50:27.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:50:27.545 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:50:29.021 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:29.018+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:50:29.550 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:29.026+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:50:29.550 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:50:29.026+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:50:37.545 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:50:37.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:50:37.555 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:50:37.556 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:50:38.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:50:38.801 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:50:38.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437438 2023-09-23T02:50:38.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437438 -le 1695440506 ]] 2023-09-23T02:50:38.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:50:38.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:50:38.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:50:48.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:50:48.805 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:50:48.806 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:50:48.810 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:50:50.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:50:50.101 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:50:50.105 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437450 2023-09-23T02:50:50.105 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437450 -le 1695440506 ]] 2023-09-23T02:50:50.105 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:50:50.105 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:50:50.105 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:00.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:00.111 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:00.111 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:00.111 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:51:01.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2953468242 2023-09-23T02:51:01.490 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:51:01.498 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437461 2023-09-23T02:51:01.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437461 -le 1695440506 ]] 2023-09-23T02:51:01.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2953468242 ]] 2023-09-23T02:51:01.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2953468242 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:51:01.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:11.511 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:11.512 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:11.513 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:11.521 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:51:12.683 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:51:12.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:51:12.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437472 2023-09-23T02:51:12.782 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437472 -le 1695440506 ]] 2023-09-23T02:51:12.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:51:12.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\9\5\3\4\6\8\2\4\2 ]] 2023-09-23T02:51:12.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/778467004 2023-09-23T02:51:14.480 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/778467004 until 2023-09-23T03:51:13.721549+0000 (3600 sec) 2023-09-23T02:51:14.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:51:14.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/778467004 2023-09-23T02:51:15.615 INFO:tasks.workunit.client.0.smithi002.stderr:listed 9 entries 2023-09-23T02:51:15.664 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/778467004 2023-09-23T02:51:15.664 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:25.674 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:25.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:51:25.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:25.709 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:26.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:51:26.899 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:51:26.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437486 2023-09-23T02:51:26.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437486 -le 1695440506 ]] 2023-09-23T02:51:26.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:51:26.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:51:26.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:36.902 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:36.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:51:36.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:36.913 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:38.309 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:51:38.628 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:51:38.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437498 2023-09-23T02:51:38.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437498 -le 1695440506 ]] 2023-09-23T02:51:38.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:51:38.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:51:38.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:48.312 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:48.312 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:48.312 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:48.321 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:51:49.611 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:51:49.611 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:51:49.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437509 2023-09-23T02:51:49.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437509 -le 1695440506 ]] 2023-09-23T02:51:49.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:51:49.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:51:49.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:51:59.624 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:51:59.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:51:59.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:51:59.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:52:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.005+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.005+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb74d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.005+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.005+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.009+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.009+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.021+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.021+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbab400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.025+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.023 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.025+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a1200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.025+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.025+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a1180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.029+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.029+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.033+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.033+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.037+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.037+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.041+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.041+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.046 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.049+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.046 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.049+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.053+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.053+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.057+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.057+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.061+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.061+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.065+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.065+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.073+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.073+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.077+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.077+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.081+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.081+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.081+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.085+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.084 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.085+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:00.084 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:52:00.085+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b7b7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:52:01.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:52:01.069 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:52:01.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437521 2023-09-23T02:52:01.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437521 -le 1695440506 ]] 2023-09-23T02:52:01.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:52:01.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:52:01.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:52:11.073 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:52:11.074 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:52:11.076 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:52:11.077 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:52:12.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/778467004 2023-09-23T02:52:12.116 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:52:12.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437532 2023-09-23T02:52:12.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437532 -le 1695440506 ]] 2023-09-23T02:52:12.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/778467004 ]] 2023-09-23T02:52:12.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/778467004 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:52:12.120 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:52:22.129 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:52:22.132 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:52:22.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:52:22.144 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:52:23.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2409967782 2023-09-23T02:52:23.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:52:23.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437543 2023-09-23T02:52:23.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437543 -le 1695440506 ]] 2023-09-23T02:52:23.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2409967782 ]] 2023-09-23T02:52:23.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2409967782 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\7\7\8\4\6\7\0\0\4 ]] 2023-09-23T02:52:23.647 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2409967782 2023-09-23T02:52:26.442 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2409967782 until 2023-09-23T03:52:25.653065+0000 (3600 sec) 2023-09-23T02:52:26.534 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2409967782 2023-09-23T02:52:26.545 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:52:27.671 INFO:tasks.workunit.client.0.smithi002.stderr:listed 10 entries 2023-09-23T02:52:27.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2409967782 2023-09-23T02:52:27.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:52:38.140 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:52:38.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:52:38.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:52:38.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:52:39.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2409967782 2023-09-23T02:52:39.854 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:52:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437559 2023-09-23T02:52:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437559 -le 1695440506 ]] 2023-09-23T02:52:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2409967782 ]] 2023-09-23T02:52:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2409967782 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\4\0\9\9\6\7\7\8\2 ]] 2023-09-23T02:52:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:52:49.864 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:52:49.864 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:52:49.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:52:49.867 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:52:51.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2409967782 2023-09-23T02:52:51.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:52:51.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437571 2023-09-23T02:52:51.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437571 -le 1695440506 ]] 2023-09-23T02:52:51.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2409967782 ]] 2023-09-23T02:52:51.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2409967782 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\4\0\9\9\6\7\7\8\2 ]] 2023-09-23T02:52:51.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:52:57.139 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:52:57.140 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:52:57.141 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:52:57.136+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:53:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bad7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.004+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.004+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.008+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.008+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.016+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.016+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbab600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.024+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.024+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.031 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.032+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.031 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.032+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf09c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.043 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.043 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.044+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.048+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.048+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.056+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.056+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf09400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.060+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.060+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.064+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.064+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.068+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.068+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.072+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.072+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.084+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.084+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.092+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.100+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:53:00.100+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:53:01.294 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:53:01.295 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:53:01.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:53:01.327 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:53:02.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2409967782 2023-09-23T02:53:02.401 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:53:02.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437582 2023-09-23T02:53:02.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437582 -le 1695440506 ]] 2023-09-23T02:53:02.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2409967782 ]] 2023-09-23T02:53:02.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2409967782 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\4\0\9\9\6\7\7\8\2 ]] 2023-09-23T02:53:02.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:53:12.414 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:53:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:53:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:53:12.428 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:53:13.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2409967782 2023-09-23T02:53:13.753 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:53:13.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437593 2023-09-23T02:53:13.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437593 -le 1695440506 ]] 2023-09-23T02:53:13.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2409967782 ]] 2023-09-23T02:53:13.755 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2409967782 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\4\0\9\9\6\7\7\8\2 ]] 2023-09-23T02:53:13.755 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:53:23.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:53:23.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:53:23.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:53:23.792 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:53:25.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:53:25.108 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:53:25.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437605 2023-09-23T02:53:25.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437605 -le 1695440506 ]] 2023-09-23T02:53:25.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:53:25.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\4\0\9\9\6\7\7\8\2 ]] 2023-09-23T02:53:25.109 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/3079579092 2023-09-23T02:53:27.482 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/3079579092 until 2023-09-23T03:53:26.776663+0000 (3600 sec) 2023-09-23T02:53:27.534 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:53:27.535 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/3079579092 2023-09-23T02:53:28.593 INFO:tasks.workunit.client.0.smithi002.stderr:listed 11 entries 2023-09-23T02:53:28.628 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/3079579092 2023-09-23T02:53:28.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:53:37.451 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:53:37.452 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:53:37.451+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:53:38.634 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:53:38.636 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:53:38.638 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:53:38.644 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:53:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:53:39.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:53:39.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437619 2023-09-23T02:53:39.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437619 -le 1695440506 ]] 2023-09-23T02:53:39.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:53:39.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:53:39.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:53:49.869 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:53:49.869 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:53:49.874 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:53:49.885 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:53:50.478 INFO:tasks.workunit.client.0.smithi002.stderr:2023-09-23T02:53:50.483+0000 7f04e6ffd640 0 --2- 172.21.15.2:0/992149256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04dc016040 0x7f04dc145ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-23T02:53:51.381 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:53:51.381 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:53:51.382 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437631 2023-09-23T02:53:51.382 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437631 -le 1695440506 ]] 2023-09-23T02:53:51.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:53:51.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:53:51.383 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:54:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.002+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.002+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.002+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.006+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a0280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.010+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.010+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1a0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.022+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.022+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.026+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.026+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.038+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.038+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.062+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.062+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.066+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.066+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.066 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.066 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.074+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.074+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.078+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.078+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.082+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.082+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.086+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.086+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.094+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.094+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.098+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.096 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.098+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.102+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:54:00.102+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:54:01.385 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:54:01.385 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:54:01.390 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:54:01.390 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:54:02.806 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:54:02.807 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:54:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437642 2023-09-23T02:54:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437642 -le 1695440506 ]] 2023-09-23T02:54:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:54:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:54:02.808 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:54:12.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:54:12.809 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:54:12.810 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:54:12.814 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:54:14.075 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:54:14.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:54:14.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437654 2023-09-23T02:54:14.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437654 -le 1695440506 ]] 2023-09-23T02:54:14.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:54:14.076 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:54:14.077 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:54:24.077 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:54:24.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:54:24.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:54:24.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:54:25.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3079579092 2023-09-23T02:54:25.428 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:54:25.428 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437665 2023-09-23T02:54:25.428 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437665 -le 1695440506 ]] 2023-09-23T02:54:25.429 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3079579092 ]] 2023-09-23T02:54:25.429 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3079579092 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:54:25.429 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:54:35.430 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:54:35.434 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:54:35.436 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:54:35.443 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:54:36.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:54:36.706 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:54:36.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437676 2023-09-23T02:54:36.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437676 -le 1695440506 ]] 2023-09-23T02:54:36.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:54:36.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\0\7\9\5\7\9\0\9\2 ]] 2023-09-23T02:54:36.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2003455134 2023-09-23T02:54:39.472 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2003455134 until 2023-09-23T03:54:38.819852+0000 (3600 sec) 2023-09-23T02:54:39.547 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:54:39.547 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2003455134 2023-09-23T02:54:40.333 INFO:tasks.workunit.client.0.smithi002.stderr:listed 12 entries 2023-09-23T02:54:40.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2003455134 2023-09-23T02:54:40.366 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:54:50.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:54:50.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:54:50.371 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:54:50.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:54:51.675 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:54:51.675 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:54:51.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437691 2023-09-23T02:54:51.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437691 -le 1695440506 ]] 2023-09-23T02:54:51.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:54:51.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:54:51.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.001+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.001+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5ca500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.005+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.005+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bbcb880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.013+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.013+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a1080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.021+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.021+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.029+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.029+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.041+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.041+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a1200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.045+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.045+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.069+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.069+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.081+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.081+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bb77a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.089+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.089+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bb77880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.097+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.097+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.105+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.105+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bb75d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.109+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.109+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.113+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.110 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.113+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.117+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.117+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bb76c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.129+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.129+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c047600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.129+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.129+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c047d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.133+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.133+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.134 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.137+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.134 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.137+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.142 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.145+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:00.142 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:55:00.145+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c047a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:55:01.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:55:01.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:55:01.692 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:55:01.700 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:55:03.253 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:55:03.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:55:03.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437703 2023-09-23T02:55:03.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437703 -le 1695440506 ]] 2023-09-23T02:55:03.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:55:03.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:55:03.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:13.260 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:55:13.261 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:55:13.261 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:55:13.261 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:55:14.660 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:55:14.675 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:55:14.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437714 2023-09-23T02:55:14.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437714 -le 1695440506 ]] 2023-09-23T02:55:14.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:55:14.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:55:14.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:24.684 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:55:24.685 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:55:24.688 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:55:24.701 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:55:25.848 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:55:25.856 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:55:25.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437725 2023-09-23T02:55:25.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437725 -le 1695440506 ]] 2023-09-23T02:55:25.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:55:25.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:55:25.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:35.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:55:35.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:55:35.877 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:55:35.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:55:37.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2003455134 2023-09-23T02:55:37.040 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:55:37.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437737 2023-09-23T02:55:37.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437737 -le 1695440506 ]] 2023-09-23T02:55:37.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2003455134 ]] 2023-09-23T02:55:37.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2003455134 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:55:37.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:47.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:55:47.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:55:47.045 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:55:47.051 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:55:48.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:55:48.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:55:48.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437748 2023-09-23T02:55:48.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437748 -le 1695440506 ]] 2023-09-23T02:55:48.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:55:48.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\0\3\4\5\5\1\3\4 ]] 2023-09-23T02:55:48.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/400934215 2023-09-23T02:55:50.580 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/400934215 until 2023-09-23T03:55:49.835163+0000 (3600 sec) 2023-09-23T02:55:50.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:55:50.643 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/400934215 2023-09-23T02:55:51.602 INFO:tasks.workunit.client.0.smithi002.stderr:listed 13 entries 2023-09-23T02:55:51.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/400934215 2023-09-23T02:55:51.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.208+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:55:58.214 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:55:58.215 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:55:58.215 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:55:58.215 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:55:58.215 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:55:58.215 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:55:58.212+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:56:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16aeee500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.004+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.004+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.008+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.008+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.020+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.020+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.020+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.020+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.028+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.028+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.032+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.032+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.034 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.036+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.034 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.036+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.052+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.052+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.080+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.080+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.084+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.084+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.100+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.100+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.104+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.104+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.108+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.108+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.116+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.116+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.124+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.123 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.124+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.130 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.132+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.132+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.136+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.136+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.140 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.140+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:00.141 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:56:00.140+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:56:01.659 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:01.661 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:56:01.665 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:01.678 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:03.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:56:03.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:56:03.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437763 2023-09-23T02:56:03.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437763 -le 1695440506 ]] 2023-09-23T02:56:03.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:56:03.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:56:03.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:56:13.157 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:13.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:13.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:56:13.310 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:14.484 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:56:14.484 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:56:14.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437774 2023-09-23T02:56:14.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437774 -le 1695440506 ]] 2023-09-23T02:56:14.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:56:14.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:56:14.486 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:56:24.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:24.489 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:56:24.490 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:24.493 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:26.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:56:26.240 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:56:26.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437786 2023-09-23T02:56:26.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437786 -le 1695440506 ]] 2023-09-23T02:56:26.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:56:26.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:56:26.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:56:36.251 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:36.263 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:36.266 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:56:36.270 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:37.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:56:37.572 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:56:37.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437797 2023-09-23T02:56:37.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437797 -le 1695440506 ]] 2023-09-23T02:56:37.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:56:37.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:56:37.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:56:44.084 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:56:44.084 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:56:44.085 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:56:44.086 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:56:44.086 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:56:44.086 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:56:44.086 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:56:44.086 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:56:44.087+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:56:47.576 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:47.576 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:47.576 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:47.584 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:56:48.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/400934215 2023-09-23T02:56:48.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:56:48.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437808 2023-09-23T02:56:48.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437808 -le 1695440506 ]] 2023-09-23T02:56:48.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/400934215 ]] 2023-09-23T02:56:48.935 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/400934215 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:56:48.935 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:56:58.937 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:56:58.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:56:58.948 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:56:58.965 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:00.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:00.390 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:00.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437820 2023-09-23T02:57:00.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437820 -le 1695440506 ]] 2023-09-23T02:57:00.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:00.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\0\0\9\3\4\2\1\5 ]] 2023-09-23T02:57:00.391 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1376584732 2023-09-23T02:57:01.338 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:01.338+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:57:01.339 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:01.338+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:57:01.339 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:01.338+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:57:01.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282900 image watch failed: 94289226237952, (107) Transport endpoint is not connected 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282900 handle_error: handle=94289226237952: (107) Transport endpoint is not connected 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c283800 image watch failed: 94289213425664, (107) Transport endpoint is not connected 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c283800 handle_error: handle=94289213425664: (107) Transport endpoint is not connected 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16b6323c0 handle_unwatch client blocklisted 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c282900 handle_rewatch: client blocklisted 2023-09-23T02:57:01.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16b632690 handle_unwatch client blocklisted 2023-09-23T02:57:01.625 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.626+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c283800 handle_rewatch: client blocklisted 2023-09-23T02:57:01.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16b7b2000 image watch failed: 94289227958272, (107) Transport endpoint is not connected 2023-09-23T02:57:01.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16b7b2000 handle_error: handle=94289227958272: (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c283200 image watch failed: 94289224595456, (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c283200 handle_error: handle=94289224595456: (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282600 image watch failed: 94289222955008, (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282600 handle_error: handle=94289222955008: (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282000 image watch failed: 94289222957056, (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282000 handle_error: handle=94289222957056: (107) Transport endpoint is not connected 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16bdf52c0 handle_unwatch client blocklisted 2023-09-23T02:57:01.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.630+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282000 handle_rewatch: client blocklisted 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c426000 image watch failed: 94289230510080, (107) Transport endpoint is not connected 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c426000 handle_error: handle=94289230510080: (107) Transport endpoint is not connected 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16ba1bdb0 handle_unwatch client blocklisted 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16b7b2000 handle_rewatch: client blocklisted 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16b65f8b0 handle_unwatch client blocklisted 2023-09-23T02:57:01.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c283200 handle_rewatch: client blocklisted 2023-09-23T02:57:01.631 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16bdf4c80 handle_unwatch client blocklisted 2023-09-23T02:57:01.631 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282600 handle_rewatch: client blocklisted 2023-09-23T02:57:01.631 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::watcher::RewatchRequest: 0x55c16bdf4c80 handle_unwatch client blocklisted 2023-09-23T02:57:01.631 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.634+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c426000 handle_rewatch: client blocklisted 2023-09-23T02:57:01.636 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.638+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282c00 image watch failed: 94289224589312, (107) Transport endpoint is not connected 2023-09-23T02:57:01.636 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.638+0000 7f3993a25640 -1 librbd::Watcher: 0x55c16c282c00 handle_error: handle=94289224589312: (107) Transport endpoint is not connected 2023-09-23T02:57:01.638 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.642+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16aee0500 handle_unwatch client blocklisted 2023-09-23T02:57:01.638 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.642+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c282c00 handle_rewatch: client blocklisted 2023-09-23T02:57:01.646 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.650+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16b79db00 image watch failed: 94289222956032, (107) Transport endpoint is not connected 2023-09-23T02:57:01.646 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.650+0000 7f3994226640 -1 librbd::Watcher: 0x55c16b79db00 handle_error: handle=94289222956032: (107) Transport endpoint is not connected 2023-09-23T02:57:01.646 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.650+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16bdf5040 handle_unwatch client blocklisted 2023-09-23T02:57:01.647 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.650+0000 7f3994226640 -1 librbd::Watcher: 0x55c16b79db00 handle_rewatch: client blocklisted 2023-09-23T02:57:01.647 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.650+0000 7f3994226640 -1 librbd::ManagedLock: 0x55c16b64e698 send_reacquire_lock: aborting reacquire due to invalid watch handle 2023-09-23T02:57:01.653 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.654+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16c282300 image watch failed: 94289224592384, (107) Transport endpoint is not connected 2023-09-23T02:57:01.653 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.654+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c282300 handle_error: handle=94289224592384: (107) Transport endpoint is not connected 2023-09-23T02:57:01.653 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.654+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16bdf4aa0 handle_unwatch client blocklisted 2023-09-23T02:57:01.653 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:01.654+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c282300 handle_rewatch: client blocklisted 2023-09-23T02:57:02.599 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1376584732 until 2023-09-23T03:57:01.881213+0000 (3600 sec) 2023-09-23T02:57:02.613 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.614+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.614+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3ad0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c2d09a0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3e10 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c33a840 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.614 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.615 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.615 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3ba0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.615 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c41b600 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.615 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16c40c4e0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.615 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c41af20 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.616 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.616 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c414580 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c41dc80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c41b340 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c337500 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c33a840 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c294ec0 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c415ce0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c428c80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.617 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.618 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.618+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16adaf020 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16c40c1a0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16b20ef20 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16c40c270 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.621 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c41a160 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16afc3b60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.622+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3d40 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.622 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c2d1760 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c2d09a0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c4149a0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c294d40 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c428200 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bf1ef00 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.623 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c414840 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c313640 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.626+0000 7f3994226640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bfa52c0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.626 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.630+0000 7f3994226640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bd77e60 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.626 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.630+0000 7f3994226640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.627 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.630+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.627 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.630+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bbaa000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.630 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.634+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.635 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.634+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.635 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.634+0000 7f3994226640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16c213140 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.635 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.634+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.635 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.634+0000 7f3994226640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16baae7e0 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.639 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.642+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.640 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:02.642+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bef1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:02.664 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1376584732 2023-09-23T02:57:02.669 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:57:03.722 INFO:tasks.workunit.client.0.smithi002.stderr:listed 14 entries 2023-09-23T02:57:03.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1376584732 2023-09-23T02:57:03.754 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:57:04.626 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.627 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3ee0 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.627 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c41b4a0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c414840 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c304f80 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.628 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16be8e180 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:04.629 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:04.630+0000 7f3993a25640 -1 librbd::ManagedLock: 0x55c16b64e698 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T02:57:07.288 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.290+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:57:07.292 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.290+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T02:57:07.299 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.298+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:57:07.300 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:57:07.301 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:07.302+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:57:09.759 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:09.750+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:57:09.759 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:09.750+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:57:09.759 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:09.754+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:57:11.493 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::image_watcher::NotifyLockOwner: 0x55c16c1a0100 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.494 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282c00 error requesting lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.494 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::ImageWatcher: 0x55c16c282c00 blocklisted waiting for exclusive lock 2023-09-23T02:57:11.494 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::ManagedLock: 0x55c16b1d2d18 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T02:57:11.494 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16b9a3c70 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.494 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.494+0000 7f3993a25640 -1 librbd::io::AioCompletion: 0x55c16c41adc0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.582 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::image_watcher::NotifyLockOwner: 0x55c16b9b4780 handle_notify: lock owner notification failed: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.583 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16c426000 error requesting lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.583 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16c426000 blocklisted waiting for exclusive lock 2023-09-23T02:57:11.583 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::ManagedLock: 0x55c16acf4eb8 send_acquire_lock: watcher not registered - client blocklisted 2023-09-23T02:57:11.583 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16c40c410 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:11.583 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:57:11.586+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16b20eb00 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:57:13.766 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:57:13.774 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:13.778 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:57:13.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:57:14.809 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.810+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T02:57:14.809 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.810+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T02:57:14.810 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.814+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T02:57:14.818 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.822+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T02:57:14.823 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:57:14.823 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:57:14.823 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:57:14.823 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:57:14.824 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:57:14.826+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:57:15.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:15.032 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:15.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437835 2023-09-23T02:57:15.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437835 -le 1695440506 ]] 2023-09-23T02:57:15.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:15.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:57:15.033 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:57:25.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:57:25.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:25.045 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:57:25.053 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:57:26.066 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:26.067 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:26.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437846 2023-09-23T02:57:26.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437846 -le 1695440506 ]] 2023-09-23T02:57:26.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:26.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:57:26.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:57:36.078 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:57:36.078 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:36.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:57:36.091 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:57:37.171 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:37.171 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437857 2023-09-23T02:57:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437857 -le 1695440506 ]] 2023-09-23T02:57:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:57:37.172 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:57:47.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:57:47.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:47.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:57:47.214 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:57:48.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:48.457 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:48.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437868 2023-09-23T02:57:48.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437868 -le 1695440506 ]] 2023-09-23T02:57:48.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:48.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:57:48.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:57:58.479 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:57:58.479 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:57:58.481 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:57:58.501 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:57:59.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1376584732 2023-09-23T02:57:59.725 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:57:59.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437879 2023-09-23T02:57:59.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437879 -le 1695440506 ]] 2023-09-23T02:57:59.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1376584732 ]] 2023-09-23T02:57:59.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1376584732 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:57:59.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:58:09.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:58:09.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:58:09.730 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:58:09.738 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:58:11.038 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:58:11.038 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:58:11.039 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437891 2023-09-23T02:58:11.039 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437891 -le 1695440506 ]] 2023-09-23T02:58:11.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:58:11.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\6\5\8\4\7\3\2 ]] 2023-09-23T02:58:11.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/3523592144 2023-09-23T02:58:13.633 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/3523592144 until 2023-09-23T03:58:12.926651+0000 (3600 sec) 2023-09-23T02:58:13.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:58:13.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/3523592144 2023-09-23T02:58:14.701 INFO:tasks.workunit.client.0.smithi002.stderr:listed 15 entries 2023-09-23T02:58:14.741 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/3523592144 2023-09-23T02:58:14.741 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:58:24.745 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:58:24.745 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:58:24.748 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:58:24.757 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:58:25.903 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:58:25.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:58:25.903 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437905 2023-09-23T02:58:25.903 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437905 -le 1695440506 ]] 2023-09-23T02:58:25.904 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:58:25.904 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:58:25.904 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:58:35.920 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:58:35.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:58:35.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:58:35.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:58:37.201 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:58:37.201 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:58:37.202 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437917 2023-09-23T02:58:37.202 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437917 -le 1695440506 ]] 2023-09-23T02:58:37.202 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:58:37.202 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:58:37.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:58:47.204 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:58:47.217 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:58:47.239 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:58:47.241 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:58:48.540 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:58:48.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:58:48.542 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437928 2023-09-23T02:58:48.542 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437928 -le 1695440506 ]] 2023-09-23T02:58:48.542 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:58:48.542 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:58:48.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:58:58.545 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:58:58.553 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:58:58.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:58:58.558 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:58:59.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:58:59.843 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:58:59.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437939 2023-09-23T02:58:59.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437939 -le 1695440506 ]] 2023-09-23T02:58:59.844 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:58:59.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:58:59.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:59:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.004+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.004+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b9b4780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.004+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.004+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbabc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.008+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.008+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1a0400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.012+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.012+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbaa780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.016+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.016+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.021 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.020+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.021 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.020+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c046400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.024+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.024+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.032+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.032+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.036+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.036+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.040+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.040+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.044+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.044+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.052+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.052+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.052+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c047000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.052+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c252300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.056+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.056+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.060+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.060+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.064+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.064+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c252480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.068+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.068+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c252680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.072+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.072+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c252100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.080+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.080+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c253e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.092+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T02:59:00.092+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c253d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T02:59:09.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:59:09.867 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:59:09.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:59:09.885 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:59:11.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3523592144 2023-09-23T02:59:11.004 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:59:11.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437951 2023-09-23T02:59:11.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437951 -le 1695440506 ]] 2023-09-23T02:59:11.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3523592144 ]] 2023-09-23T02:59:11.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3523592144 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:59:11.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:59:18.627 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.627+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:59:18.733 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.723+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T02:59:18.734 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.727+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T02:59:18.734 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.731+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T02:59:18.734 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.735+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T02:59:18.735 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.735+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T02:59:18.743 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.739+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T02:59:18.743 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.739+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T02:59:18.744 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.739+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T02:59:18.744 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.739+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T02:59:18.744 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T02:59:18.739+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T02:59:21.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:59:21.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:59:21.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:59:21.018 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:59:21.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T02:59:21.990 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:59:21.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437961 2023-09-23T02:59:21.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437961 -le 1695440506 ]] 2023-09-23T02:59:21.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T02:59:21.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\2\3\5\9\2\1\4\4 ]] 2023-09-23T02:59:21.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/312273758 2023-09-23T02:59:24.644 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/312273758 until 2023-09-23T03:59:23.967768+0000 (3600 sec) 2023-09-23T02:59:24.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/312273758 2023-09-23T02:59:24.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T02:59:25.583 INFO:tasks.workunit.client.0.smithi002.stderr:listed 16 entries 2023-09-23T02:59:25.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/312273758 2023-09-23T02:59:25.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:59:35.626 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:59:35.626 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:59:35.633 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:59:35.650 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:59:36.923 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T02:59:36.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:59:36.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437976 2023-09-23T02:59:36.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437976 -le 1695440506 ]] 2023-09-23T02:59:36.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T02:59:36.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T02:59:36.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:59:46.939 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:59:46.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:59:46.948 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:59:46.972 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:59:48.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T02:59:48.496 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:59:48.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437988 2023-09-23T02:59:48.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437988 -le 1695440506 ]] 2023-09-23T02:59:48.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T02:59:48.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T02:59:48.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T02:59:58.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T02:59:58.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T02:59:58.505 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T02:59:58.519 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T02:59:59.912 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T02:59:59.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T02:59:59.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695437999 2023-09-23T02:59:59.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695437999 -le 1695440506 ]] 2023-09-23T02:59:59.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T02:59:59.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T02:59:59.913 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:00:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.003+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.003+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5dab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.003+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.003+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b7b6380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.011+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.011+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16addc800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.015+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.015+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b7b6680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.035+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.035+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bb75e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.047+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.047+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bbabc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.059+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.059+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5cab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.075+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.075+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aef2880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.087+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.087+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bbaa480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.087+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.087+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aeef880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.099+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.099+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aeee700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.103+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.103+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5cb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.107+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.103 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.107+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a1080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.107 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.111+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.107 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.111+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.115+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.115+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.119+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.119+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c074f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.127+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.127+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bef0880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.131+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.131+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bef1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.135+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.135+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c046e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.139+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:00:00.139+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bef1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:00:00.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:00:00.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:00:00.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:00:00.887 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:00:00.888 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:00:00.889 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:00:00.890 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:00:00.887+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:00:09.915 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:00:09.915 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:00:09.924 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:00:09.925 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:00:11.146 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T03:00:11.146 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:00:11.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438011 2023-09-23T03:00:11.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438011 -le 1695440506 ]] 2023-09-23T03:00:11.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T03:00:11.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T03:00:11.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:00:21.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:00:21.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:00:21.157 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:00:21.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:00:22.485 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/312273758 2023-09-23T03:00:22.494 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:00:22.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438022 2023-09-23T03:00:22.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438022 -le 1695440506 ]] 2023-09-23T03:00:22.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/312273758 ]] 2023-09-23T03:00:22.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/312273758 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T03:00:22.496 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:00:32.511 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:00:32.512 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:00:32.512 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:00:32.517 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:00:33.981 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:00:33.986 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:00:33.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438033 2023-09-23T03:00:33.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438033 -le 1695440506 ]] 2023-09-23T03:00:33.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:00:33.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\1\2\2\7\3\7\5\8 ]] 2023-09-23T03:00:33.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2045148581 2023-09-23T03:00:35.688 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2045148581 until 2023-09-23T04:00:34.926008+0000 (3600 sec) 2023-09-23T03:00:35.729 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2045148581 2023-09-23T03:00:35.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:00:36.833 INFO:tasks.workunit.client.0.smithi002.stderr:listed 17 entries 2023-09-23T03:00:36.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2045148581 2023-09-23T03:00:36.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:00:46.888 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:00:46.888 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:00:46.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:00:46.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:00:48.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:00:48.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:00:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438048 2023-09-23T03:00:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438048 -le 1695440506 ]] 2023-09-23T03:00:48.148 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:00:48.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:00:48.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:00:58.151 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:00:58.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:00:58.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:00:58.185 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:00:59.224 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:00:59.233 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:00:59.234 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438059 2023-09-23T03:00:59.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438059 -le 1695440506 ]] 2023-09-23T03:00:59.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:00:59.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:00:59.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:01:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.006+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.006+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c252400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.006+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.006+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be93380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.022+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.022+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b7b6280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.026+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.026+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.046+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.046+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.050+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.050+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.054+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.054+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c075000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.066+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.066+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c075100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.078+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.078+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.082+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.082+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.086+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.086+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.090+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.090+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c075280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.094+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.094+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.098+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.102+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.106+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.106+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.110+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.110+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.118+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:01:00.118+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:01:09.240 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:01:09.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:01:09.243 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:01:09.254 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:01:10.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:01:10.413 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:01:10.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438070 2023-09-23T03:01:10.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438070 -le 1695440506 ]] 2023-09-23T03:01:10.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:01:10.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:01:10.415 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:01:20.428 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:01:20.446 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:01:20.449 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:01:20.453 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:01:21.691 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:01:21.691 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:01:21.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438081 2023-09-23T03:01:21.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438081 -le 1695440506 ]] 2023-09-23T03:01:21.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:01:21.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:01:21.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:01:31.698 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:01:31.698 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:01:31.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:01:31.721 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:01:32.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2045148581 2023-09-23T03:01:33.014 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:01:33.015 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438093 2023-09-23T03:01:33.015 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438093 -le 1695440506 ]] 2023-09-23T03:01:33.015 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2045148581 ]] 2023-09-23T03:01:33.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2045148581 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:01:33.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:01:43.017 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:01:43.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:01:43.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:01:43.022 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:01:44.304 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:01:44.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:01:44.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438104 2023-09-23T03:01:44.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438104 -le 1695440506 ]] 2023-09-23T03:01:44.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:01:44.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\0\4\5\1\4\8\5\8\1 ]] 2023-09-23T03:01:44.312 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/4137551135 2023-09-23T03:01:46.652 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/4137551135 until 2023-09-23T04:01:45.936529+0000 (3600 sec) 2023-09-23T03:01:46.689 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:01:46.689 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/4137551135 2023-09-23T03:01:47.607 INFO:tasks.workunit.client.0.smithi002.stderr:listed 18 entries 2023-09-23T03:01:47.683 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/4137551135 2023-09-23T03:01:47.683 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:01:57.694 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:01:57.705 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:01:57.712 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:01:57.721 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:01:59.140 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:01:59.140 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:01:59.141 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438119 2023-09-23T03:01:59.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438119 -le 1695440506 ]] 2023-09-23T03:01:59.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:01:59.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:01:59.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.005+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.005+0000 7f3994226640 -1 librbd::ImageState: 0x55c16addc880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.009+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.009+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb76300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.009+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.009+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c252380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.017+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.017+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bb77000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.029+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.029+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c252b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.029+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.029+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.033 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.037+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.033 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.037+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.045+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.045+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.049+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.049+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.053+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.053+0000 7f3994226640 -1 librbd::ImageState: 0x55c16aef2f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.061+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.061+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b4a0580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.065+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.065+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b5cb600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.070 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.073+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.073+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.081+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.081+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.101+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.101+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.109+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.109+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.121+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.121+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.130 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.133+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.133+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fcc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.137+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.137+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.140 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.141+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:00.140 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:02:00.141+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:09.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:02:09.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:02:09.144 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:02:09.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:02:10.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:02:10.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:02:10.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438130 2023-09-23T03:02:10.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438130 -le 1695440506 ]] 2023-09-23T03:02:10.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:02:10.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:02:10.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:20.559 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:02:20.565 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:02:20.587 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:02:20.590 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:02:22.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:02:22.043 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:02:22.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438142 2023-09-23T03:02:22.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438142 -le 1695440506 ]] 2023-09-23T03:02:22.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:02:22.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:02:22.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:32.045 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:02:32.046 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:02:32.046 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:02:32.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:02:33.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:02:33.135 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:02:33.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438153 2023-09-23T03:02:33.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438153 -le 1695440506 ]] 2023-09-23T03:02:33.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:02:33.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:02:33.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:34.668 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:02:34.668+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:02:35.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:02:34.672+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:02:35.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:02:34.672+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:02:43.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:02:43.147 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:02:43.157 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:02:43.163 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:02:44.390 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4137551135 2023-09-23T03:02:44.402 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:02:44.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438164 2023-09-23T03:02:44.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438164 -le 1695440506 ]] 2023-09-23T03:02:44.409 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4137551135 ]] 2023-09-23T03:02:44.410 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4137551135 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:02:44.410 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:51.484 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:02:51.480+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:02:51.492 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:02:51.492+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:02:54.423 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:02:54.426 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:02:54.429 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:02:54.443 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:02:55.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:02:55.670 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:02:55.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438175 2023-09-23T03:02:55.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438175 -le 1695440506 ]] 2023-09-23T03:02:55.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:02:55.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\3\7\5\5\1\1\3\5 ]] 2023-09-23T03:02:55.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/4140630340 2023-09-23T03:02:58.311 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/4140630340 until 2023-09-23T04:02:57.652843+0000 (3600 sec) 2023-09-23T03:02:58.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/4140630340 2023-09-23T03:02:58.355 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:02:59.464 INFO:tasks.workunit.client.0.smithi002.stderr:listed 19 entries 2023-09-23T03:02:59.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/4140630340 2023-09-23T03:02:59.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:02:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.000+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:02:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.000+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aeef700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.000+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.000+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5dab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.004+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.004+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bb75c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.008+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.008+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bbaa100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.016+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.016+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bb75080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.020+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.020+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c252580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.028+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.028+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf08800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.032+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.030 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.032+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.052+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.052+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5cba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.056+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.056+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5cac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.060+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.060+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bef1f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.068+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.068+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bf09f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.088+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.088+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1a1180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.100+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.100+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b7b6000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.110 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.112+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.110 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.112+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b5cab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.116+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.114 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.116+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c075900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.120+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.120+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.125 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.128+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.125 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.128+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a0200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.132+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.132+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c046100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.134 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.136+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:03:00.136+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c046400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:03:07.754 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:03:07.755 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:07.756+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:03:09.513 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:03:09.513 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:03:09.514 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:03:09.514 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:03:10.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:03:10.821 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:03:10.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438190 2023-09-23T03:03:10.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438190 -le 1695440506 ]] 2023-09-23T03:03:10.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:03:10.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:03:10.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:03:20.828 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:03:20.828 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:03:20.828 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:03:20.845 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:03:22.104 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:03:22.119 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:03:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438202 2023-09-23T03:03:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438202 -le 1695440506 ]] 2023-09-23T03:03:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:03:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:03:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:03:32.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:03:32.145 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:03:32.146 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:03:32.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:03:33.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:03:33.361 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:03:33.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438213 2023-09-23T03:03:33.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438213 -le 1695440506 ]] 2023-09-23T03:03:33.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:03:33.362 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:03:33.362 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:03:43.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:03:43.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:03:43.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:03:43.378 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:03:44.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:03:44.593 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:03:44.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438224 2023-09-23T03:03:44.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438224 -le 1695440506 ]] 2023-09-23T03:03:44.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:03:44.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:03:44.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:03:53.231 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:53.223+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:03:53.233 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:53.235+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:03:53.251 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:53.247+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:03:53.252 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:03:53.247+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:03:54.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:03:54.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:03:54.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:03:54.600 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:03:55.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4140630340 2023-09-23T03:03:55.819 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:03:55.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438235 2023-09-23T03:03:55.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438235 -le 1695440506 ]] 2023-09-23T03:03:55.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4140630340 ]] 2023-09-23T03:03:55.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4140630340 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:03:55.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:04:05.833 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:04:05.833 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:04:05.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:04:05.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:04:07.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:04:07.094 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:04:07.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438247 2023-09-23T03:04:07.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438247 -le 1695440506 ]] 2023-09-23T03:04:07.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:04:07.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\4\0\6\3\0\3\4\0 ]] 2023-09-23T03:04:07.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1286535523 2023-09-23T03:04:09.677 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1286535523 until 2023-09-23T04:04:08.920540+0000 (3600 sec) 2023-09-23T03:04:09.718 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:04:09.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1286535523 2023-09-23T03:04:10.779 INFO:tasks.workunit.client.0.smithi002.stderr:listed 20 entries 2023-09-23T03:04:10.847 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1286535523 2023-09-23T03:04:10.847 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:04:20.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:04:20.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:04:20.864 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:04:20.886 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:04:21.873 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:04:21.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:04:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438261 2023-09-23T03:04:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438261 -le 1695440506 ]] 2023-09-23T03:04:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:04:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:04:21.877 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:04:31.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:04:31.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:04:31.894 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:04:31.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:04:32.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:04:32.985 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:04:32.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438272 2023-09-23T03:04:32.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438272 -le 1695440506 ]] 2023-09-23T03:04:32.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:04:32.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:04:32.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:04:42.994 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:04:42.994 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:04:42.995 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:04:43.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:04:44.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:04:44.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:04:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438284 2023-09-23T03:04:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438284 -le 1695440506 ]] 2023-09-23T03:04:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:04:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:04:44.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:04:54.366 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:04:54.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:04:54.390 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:04:54.393 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:04:55.005 INFO:tasks.workunit.client.0.smithi002.stderr:2023-09-23T03:04:55.009+0000 7f3785506640 0 --2- 172.21.15.2:0/253317575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37801274d0 0x7f378012a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-23T03:04:55.941 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:04:55.943 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:04:55.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438295 2023-09-23T03:04:55.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438295 -le 1695440506 ]] 2023-09-23T03:04:55.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:04:55.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:04:55.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:05:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.001+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.001+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aeef700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.005+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.005+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bbcac80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.009+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.009+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c075c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.013+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.013+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c075200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.021+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.021+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.025+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.025+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.029+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.029+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c046b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.033+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.033+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.041+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.041+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.049+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.049+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fcc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.061+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.061+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fcc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.077+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.077+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c046b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.085+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.083 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.085+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fc980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.093+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.093 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.097+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fd180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.105+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.105+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.113+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.113+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.117+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.115 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.117+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.125+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.125+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fdb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.125+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.124 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.125+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fdd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.133+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:00.133+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fdf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:05:05.952 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:05:05.954 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:05:05.960 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:05:05.967 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:05:07.173 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1286535523 2023-09-23T03:05:07.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:05:07.175 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438307 2023-09-23T03:05:07.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438307 -le 1695440506 ]] 2023-09-23T03:05:07.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1286535523 ]] 2023-09-23T03:05:07.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1286535523 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:05:07.176 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:05:17.181 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:05:17.188 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:05:17.188 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:05:17.190 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:05:18.557 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:05:18.558 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:05:18.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438318 2023-09-23T03:05:18.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438318 -le 1695440506 ]] 2023-09-23T03:05:18.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:05:18.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\2\8\6\5\3\5\5\2\3 ]] 2023-09-23T03:05:18.560 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1377367264 2023-09-23T03:05:20.746 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1377367264 until 2023-09-23T04:05:20.131785+0000 (3600 sec) 2023-09-23T03:05:20.812 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:05:20.813 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1377367264 2023-09-23T03:05:21.838 INFO:tasks.workunit.client.0.smithi002.stderr:listed 21 entries 2023-09-23T03:05:21.901 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1377367264 2023-09-23T03:05:21.902 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:05:31.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:05:31.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:05:31.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:05:31.915 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:05:33.110 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:05:33.110 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:05:33.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438333 2023-09-23T03:05:33.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438333 -le 1695440506 ]] 2023-09-23T03:05:33.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:05:33.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:05:33.111 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:05:43.114 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:05:43.114 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:05:43.116 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:05:43.143 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:05:44.211 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:05:44.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:05:44.220 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438344 2023-09-23T03:05:44.221 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438344 -le 1695440506 ]] 2023-09-23T03:05:44.221 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:05:44.221 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:05:44.221 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:05:54.239 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:05:54.240 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:05:54.249 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:05:54.255 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:05:55.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:05:55.489 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:05:55.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438355 2023-09-23T03:05:55.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438355 -le 1695440506 ]] 2023-09-23T03:05:55.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:05:55.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:05:55.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:59.999+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:05:59.999+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c047f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b1fdd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b1fde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.011+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b1fdc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.019+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.019+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a0a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.039+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.047+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.051+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.055+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.055+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.059+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.059+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.063+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.063+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee4200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.067+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.067+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee5200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.071+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.071+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.095+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.103+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.103+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.107+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.107+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee4a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.127+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.127+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee4080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.134 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.135+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.134 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.135+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.138 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.139+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:00.138 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:06:00.139+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:05.493 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:06:05.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:06:05.508 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:06:05.529 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:06:06.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:06:06.986 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:06:06.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438366 2023-09-23T03:06:06.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438366 -le 1695440506 ]] 2023-09-23T03:06:06.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:06:06.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:06:06.988 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:17.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:06:17.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:06:17.013 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:06:17.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:06:18.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1377367264 2023-09-23T03:06:18.117 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:06:18.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438378 2023-09-23T03:06:18.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438378 -le 1695440506 ]] 2023-09-23T03:06:18.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1377367264 ]] 2023-09-23T03:06:18.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1377367264 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:06:18.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:28.121 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:06:28.121 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:06:28.125 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:06:28.130 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:06:29.405 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:06:29.406 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:06:29.406 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438389 2023-09-23T03:06:29.406 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438389 -le 1695440506 ]] 2023-09-23T03:06:29.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:06:29.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\3\7\7\3\6\7\2\6\4 ]] 2023-09-23T03:06:29.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2866803494 2023-09-23T03:06:31.761 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2866803494 until 2023-09-23T04:06:31.084138+0000 (3600 sec) 2023-09-23T03:06:31.817 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2866803494 2023-09-23T03:06:31.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:06:32.965 INFO:tasks.workunit.client.0.smithi002.stderr:listed 22 entries 2023-09-23T03:06:32.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2866803494 2023-09-23T03:06:32.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:43.001 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:06:43.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:06:43.011 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:06:43.018 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:06:44.167 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:06:44.167 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:06:44.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438404 2023-09-23T03:06:44.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438404 -le 1695440506 ]] 2023-09-23T03:06:44.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:06:44.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:06:44.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:54.173 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:06:54.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:06:54.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:06:54.182 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:06:55.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:06:55.592 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:06:55.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438415 2023-09-23T03:06:55.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438415 -le 1695440506 ]] 2023-09-23T03:06:55.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:06:55.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:06:55.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:06:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.001+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:06:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.001+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fd180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.005+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.005+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c047200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.009+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.009+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aef2e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.017+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.017+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.017+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c046480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.017+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a0f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.021+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.021+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.045+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.045+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.045+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.045+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.057+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.057+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.069+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.069+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.081+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.081 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.081+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.089+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.089+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b4a1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.109+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.109+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.121+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.121+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.139 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.141+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.139 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.141+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.140 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.141+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.140 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.141+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.146 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.149+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.146 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.149+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.151 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.153+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.151 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.153+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.156 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.157+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.157 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.157+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed8380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.162 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.165+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:00.163 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:00.165+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed8580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:07:05.605 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:07:05.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:07:05.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:07:05.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:07:06.865 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:07:06.870 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:07:06.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438426 2023-09-23T03:07:06.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438426 -le 1695440506 ]] 2023-09-23T03:07:06.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:07:06.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:07:06.872 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:07:16.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:07:16.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:07:16.874 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:07:16.874 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:07:18.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:07:18.243 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:07:18.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438438 2023-09-23T03:07:18.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438438 -le 1695440506 ]] 2023-09-23T03:07:18.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:07:18.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:07:18.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:07:28.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:07:28.258 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:07:28.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:07:28.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:07:29.712 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2866803494 2023-09-23T03:07:29.712 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:07:29.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438449 2023-09-23T03:07:29.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438449 -le 1695440506 ]] 2023-09-23T03:07:29.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2866803494 ]] 2023-09-23T03:07:29.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2866803494 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:07:29.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:07:34.668 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.656+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:07:34.669 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:07:34.670 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:34.660+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:07:39.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:07:39.730 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:07:39.734 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:07:39.754 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:07:41.094 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:07:41.094 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:07:41.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438461 2023-09-23T03:07:41.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438461 -le 1695440506 ]] 2023-09-23T03:07:41.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:07:41.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\6\6\8\0\3\4\9\4 ]] 2023-09-23T03:07:41.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/3873857838 2023-09-23T03:07:43.817 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/3873857838 until 2023-09-23T04:07:43.253303+0000 (3600 sec) 2023-09-23T03:07:43.887 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:07:43.888 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/3873857838 2023-09-23T03:07:45.030 INFO:tasks.workunit.client.0.smithi002.stderr:listed 23 entries 2023-09-23T03:07:45.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/3873857838 2023-09-23T03:07:45.114 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:07:48.446 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:48.444+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:07:48.453 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:07:48.452+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:07:55.129 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:07:55.130 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:07:55.130 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:07:55.133 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:07:56.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:07:56.432 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:07:56.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438476 2023-09-23T03:07:56.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438476 -le 1695440506 ]] 2023-09-23T03:07:56.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:07:56.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:07:56.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:08:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:59.999+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:59.999+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b5cb500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:59.999+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:07:59.999+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.011+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.011+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c252d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.015+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.015+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.027+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.027+0000 7f3994226640 -1 librbd::ImageState: 0x55c16aef3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.039+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.039+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fdc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.039+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.039+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.043+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.043+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c075580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.051+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.051+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a1d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.071+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.071+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.071+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.074 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.071+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbaa080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.075+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.075+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.079+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.079+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.083 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.083+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.083 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.083+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.083+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.083+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bef0900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.087+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.087+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bed8680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.091+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.091+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bed8980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.095+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.095+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bed9280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.107+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:00.107 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:08:00.107+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bed9100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:08:06.439 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:08:06.442 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:08:06.450 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:08:06.450 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:08:07.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:08:07.795 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:08:07.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438487 2023-09-23T03:08:07.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438487 -le 1695440506 ]] 2023-09-23T03:08:07.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:08:07.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:08:07.797 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:08:17.799 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:08:17.799 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:08:17.799 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:08:17.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:08:18.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:08:18.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:08:18.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438498 2023-09-23T03:08:18.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438498 -le 1695440506 ]] 2023-09-23T03:08:18.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:08:18.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:08:18.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:08:28.924 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:08:28.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:08:28.930 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:08:28.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:08:30.051 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:08:30.051 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:08:30.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438510 2023-09-23T03:08:30.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438510 -le 1695440506 ]] 2023-09-23T03:08:30.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:08:30.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:08:30.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:08:30.193 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.191+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:08:30.194 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.191+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:08:30.200 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:08:30.201 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:08:30.201 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:08:30.201 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:08:30.201 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:08:30.201 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:30.195+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:08:40.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:08:40.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:08:40.056 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:08:40.057 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:08:41.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3873857838 2023-09-23T03:08:41.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:08:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438521 2023-09-23T03:08:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438521 -le 1695440506 ]] 2023-09-23T03:08:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3873857838 ]] 2023-09-23T03:08:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3873857838 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:08:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:08:47.075 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:08:47.074+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:08:51.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:08:51.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:08:51.382 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:08:51.400 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:08:52.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1878591735 2023-09-23T03:08:52.678 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:08:52.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438532 2023-09-23T03:08:52.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438532 -le 1695440506 ]] 2023-09-23T03:08:52.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1878591735 ]] 2023-09-23T03:08:52.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1878591735 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\8\7\3\8\5\7\8\3\8 ]] 2023-09-23T03:08:52.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1878591735 2023-09-23T03:08:54.824 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1878591735 until 2023-09-23T04:08:54.174286+0000 (3600 sec) 2023-09-23T03:08:54.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:08:54.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1878591735 2023-09-23T03:08:55.759 INFO:tasks.workunit.client.0.smithi002.stderr:listed 24 entries 2023-09-23T03:08:55.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1878591735 2023-09-23T03:08:55.810 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.002+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.002+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed9400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.002+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.002+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.010+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.010+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed9800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.022+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.022+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed9a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.023 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.026+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.023 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.026+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bed9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.026+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.026+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.031 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.034+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.031 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.034+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bed9f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.034+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.034+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.038+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.038+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c01a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c01a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.046+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.046 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.046+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.050+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.047 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.050+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.054+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.054+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01a900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.058+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.058+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01ab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.062+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.062+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01ad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.066+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.066+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.070+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.070+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.082+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.082+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.090+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.087 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.090+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01b500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.090+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:09:00.090+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01b700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:05.811 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:09:05.822 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:09:05.822 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:09:05.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:09:07.174 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1878591735 2023-09-23T03:09:07.175 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:09:07.177 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438547 2023-09-23T03:09:07.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438547 -le 1695440506 ]] 2023-09-23T03:09:07.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1878591735 ]] 2023-09-23T03:09:07.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1878591735 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\8\7\8\5\9\1\7\3\5 ]] 2023-09-23T03:09:07.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:17.187 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:09:17.187 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:09:17.188 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:09:17.194 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:09:18.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1878591735 2023-09-23T03:09:18.708 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:09:18.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438558 2023-09-23T03:09:18.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438558 -le 1695440506 ]] 2023-09-23T03:09:18.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1878591735 ]] 2023-09-23T03:09:18.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1878591735 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\8\7\8\5\9\1\7\3\5 ]] 2023-09-23T03:09:18.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:28.711 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:09:28.711 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:09:28.712 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:09:28.731 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:09:29.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1878591735 2023-09-23T03:09:29.918 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:09:29.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438569 2023-09-23T03:09:29.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438569 -le 1695440506 ]] 2023-09-23T03:09:29.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1878591735 ]] 2023-09-23T03:09:29.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1878591735 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\8\7\8\5\9\1\7\3\5 ]] 2023-09-23T03:09:29.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:39.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:09:39.927 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:09:39.935 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:09:39.937 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:09:41.359 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1878591735 2023-09-23T03:09:41.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:09:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438581 2023-09-23T03:09:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438581 -le 1695440506 ]] 2023-09-23T03:09:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1878591735 ]] 2023-09-23T03:09:41.370 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1878591735 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\8\7\8\5\9\1\7\3\5 ]] 2023-09-23T03:09:41.371 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:51.382 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:09:51.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:09:51.387 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:09:51.398 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:09:52.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:09:52.634 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:09:52.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438592 2023-09-23T03:09:52.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438592 -le 1695440506 ]] 2023-09-23T03:09:52.640 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:09:52.641 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\8\7\8\5\9\1\7\3\5 ]] 2023-09-23T03:09:52.641 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1770443901 2023-09-23T03:09:53.716 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:53.717+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:09:53.724 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:53.721+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:09:53.724 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:53.721+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:09:54.875 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1770443901 until 2023-09-23T04:09:54.104077+0000 (3600 sec) 2023-09-23T03:09:54.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:09:54.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1770443901 2023-09-23T03:09:55.615 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.613+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:09:55.615 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.613+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:09:55.622 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:09:55.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:09:55.624 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:09:55.617+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:09:56.043 INFO:tasks.workunit.client.0.smithi002.stderr:listed 25 entries 2023-09-23T03:09:56.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1770443901 2023-09-23T03:09:56.093 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:09:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.000+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:09:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.000+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c074680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c01aa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.012+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.012+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bf08100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.016+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.016+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.020+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.020+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fe280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.036+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.036+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fe480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.040+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.040+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fe680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.053 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.052+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.053 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.052+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fe880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.056+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.056+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.060+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.060+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.064+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.064+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3fee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.080+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.080+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.080+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.080+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.092+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ffe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.096+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.096+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ffd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.104+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.104+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ffb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.108+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.108+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.112+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:10:00.112+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:10:06.100 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:10:06.101 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:10:06.101 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:10:06.117 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:10:07.406 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:10:07.406 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:10:07.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438607 2023-09-23T03:10:07.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438607 -le 1695440506 ]] 2023-09-23T03:10:07.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:10:07.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:10:07.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:10:09.706 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:09.704+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:10:09.706 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:09.704+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:10:17.414 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:10:17.420 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:10:17.425 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:10:17.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:10:18.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:10:18.801 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:10:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438618 2023-09-23T03:10:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438618 -le 1695440506 ]] 2023-09-23T03:10:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:10:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:10:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:10:28.821 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:10:28.825 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:10:28.825 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:10:28.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:10:29.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:10:29.857 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:10:29.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438629 2023-09-23T03:10:29.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438629 -le 1695440506 ]] 2023-09-23T03:10:29.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:10:29.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:10:29.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:10:36.037 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:10:36.038 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:10:36.039 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:36.036+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:10:39.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:10:39.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:10:39.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:10:39.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:10:41.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:10:41.204 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:10:41.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438641 2023-09-23T03:10:41.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438641 -le 1695440506 ]] 2023-09-23T03:10:41.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:10:41.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:10:41.205 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:10:44.449 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:44.439+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:10:46.151 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:10:46.152 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:10:46.147+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:10:51.211 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:10:51.211 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:10:51.211 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:10:51.217 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:10:52.633 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:10:52.648 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:10:52.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438652 2023-09-23T03:10:52.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438652 -le 1695440506 ]] 2023-09-23T03:10:52.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:10:52.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:10:52.649 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:02.653 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:11:02.653 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:11:02.654 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:11:02.679 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:11:03.915 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1770443901 2023-09-23T03:11:03.916 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:11:03.917 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438663 2023-09-23T03:11:03.917 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438663 -le 1695440506 ]] 2023-09-23T03:11:03.917 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1770443901 ]] 2023-09-23T03:11:03.917 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1770443901 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:11:03.918 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:13.919 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:11:13.920 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:11:13.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:11:13.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:11:15.090 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:11:15.091 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:11:15.091 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438675 2023-09-23T03:11:15.091 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438675 -le 1695440506 ]] 2023-09-23T03:11:15.091 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:11:15.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\7\7\0\4\4\3\9\0\1 ]] 2023-09-23T03:11:15.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/860360033 2023-09-23T03:11:17.884 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/860360033 until 2023-09-23T04:11:17.170561+0000 (3600 sec) 2023-09-23T03:11:17.942 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/860360033 2023-09-23T03:11:17.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:11:19.010 INFO:tasks.workunit.client.0.smithi002.stderr:listed 26 entries 2023-09-23T03:11:19.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/860360033 2023-09-23T03:11:19.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:29.077 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:11:29.077 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:11:29.081 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:11:29.107 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:11:30.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:11:30.328 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:11:30.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438690 2023-09-23T03:11:30.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438690 -le 1695440506 ]] 2023-09-23T03:11:30.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:11:30.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:11:30.329 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:40.333 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:11:40.333 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:11:40.342 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:11:40.342 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:11:41.475 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:11:41.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:11:41.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438701 2023-09-23T03:11:41.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438701 -le 1695440506 ]] 2023-09-23T03:11:41.477 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:11:41.477 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:11:41.477 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:46.708 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:11:46.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:11:46.710 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:11:46.706+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:11:51.485 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:11:51.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:11:51.490 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:11:51.498 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:11:52.725 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:11:52.725 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:11:52.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438712 2023-09-23T03:11:52.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438712 -le 1695440506 ]] 2023-09-23T03:11:52.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:11:52.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:11:52.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:11:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.002+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:11:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.002+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c047900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.002+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.002+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c252b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.006+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.006+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1a0080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.014+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.014+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.026+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.026 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.026+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee5680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.054+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.054+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.066+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.066+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.074+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.074+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.078+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.078+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.082+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.079 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.082+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.086+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.086+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.086+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.086+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.094+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.094+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.094+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.094+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.098+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.098+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.102+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:12:00.102+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:12:02.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:12:02.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:12:02.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:12:02.739 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:12:03.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:12:03.873 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:12:03.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438723 2023-09-23T03:12:03.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438723 -le 1695440506 ]] 2023-09-23T03:12:03.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:12:03.874 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:12:03.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:12:13.877 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:12:13.877 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:12:13.878 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:12:13.881 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:12:15.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/860360033 2023-09-23T03:12:15.352 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:12:15.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438735 2023-09-23T03:12:15.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438735 -le 1695440506 ]] 2023-09-23T03:12:15.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/860360033 ]] 2023-09-23T03:12:15.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/860360033 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:12:15.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:12:25.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:12:25.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:12:25.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:12:25.374 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:12:26.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:12:26.465 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:12:26.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438746 2023-09-23T03:12:26.465 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438746 -le 1695440506 ]] 2023-09-23T03:12:26.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:12:26.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\8\6\0\3\6\0\0\3\3 ]] 2023-09-23T03:12:26.466 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1510231876 2023-09-23T03:12:28.929 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1510231876 until 2023-09-23T04:12:28.195881+0000 (3600 sec) 2023-09-23T03:12:28.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:12:28.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1510231876 2023-09-23T03:12:30.112 INFO:tasks.workunit.client.0.smithi002.stderr:listed 27 entries 2023-09-23T03:12:30.159 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1510231876 2023-09-23T03:12:30.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:12:40.162 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:12:40.163 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:12:40.167 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:12:40.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:12:41.460 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:12:41.460 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:12:41.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438761 2023-09-23T03:12:41.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438761 -le 1695440506 ]] 2023-09-23T03:12:41.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:12:41.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:12:41.461 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:12:51.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:12:51.467 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:12:51.471 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:12:51.471 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:12:52.760 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:12:52.760 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:12:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438772 2023-09-23T03:12:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438772 -le 1695440506 ]] 2023-09-23T03:12:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:12:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:12:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:13:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.001+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.001+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1a0f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.005+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.005+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ff700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.013+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.013+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1c3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.025+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.025+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ff880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.039 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1c3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aa180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.041+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aa380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.046 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.045+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.046 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.045+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aa580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.053+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.053+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aa780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.057+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.057+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aa980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.061+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.061+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.065+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.065+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.069+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.069+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aaf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.073+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.073+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ab180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.077+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.078 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.077+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ab380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.089+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.089+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.089+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3abb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.089+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.093+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.093+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ab800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.096 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.097+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.096 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.097+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ab580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.101+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:13:00.101+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3abd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:13:02.764 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:13:02.764 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:13:02.771 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:13:02.774 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:13:04.115 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:13:04.115 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:13:04.115 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438784 2023-09-23T03:13:04.115 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438784 -le 1695440506 ]] 2023-09-23T03:13:04.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:13:04.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:13:04.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:13:14.118 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:13:14.119 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:13:14.120 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:13:14.129 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:13:15.429 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:13:15.429 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:13:15.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438795 2023-09-23T03:13:15.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438795 -le 1695440506 ]] 2023-09-23T03:13:15.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:13:15.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:13:15.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:13:25.448 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:13:25.457 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:13:25.457 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:13:25.459 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:13:26.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1510231876 2023-09-23T03:13:26.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:13:26.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438806 2023-09-23T03:13:26.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438806 -le 1695440506 ]] 2023-09-23T03:13:26.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1510231876 ]] 2023-09-23T03:13:26.829 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1510231876 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:13:26.829 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:13:36.834 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:13:36.839 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:13:36.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:13:36.864 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:13:38.157 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:13:38.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:13:38.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438818 2023-09-23T03:13:38.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438818 -le 1695440506 ]] 2023-09-23T03:13:38.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:13:38.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\5\1\0\2\3\1\8\7\6 ]] 2023-09-23T03:13:38.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/3586892185 2023-09-23T03:13:39.928 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/3586892185 until 2023-09-23T04:13:39.164664+0000 (3600 sec) 2023-09-23T03:13:39.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:13:40.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/3586892185 2023-09-23T03:13:40.961 INFO:tasks.workunit.client.0.smithi002.stderr:listed 28 entries 2023-09-23T03:13:41.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/3586892185 2023-09-23T03:13:41.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:13:51.039 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:13:51.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:13:51.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:13:51.056 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:13:52.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:13:52.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:13:52.263 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438832 2023-09-23T03:13:52.263 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438832 -le 1695440506 ]] 2023-09-23T03:13:52.263 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:13:52.263 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:13:52.263 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:14:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.004+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.004+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.004+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1c3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.004+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ab700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.008+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.008+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ab980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.020+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.020+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3aae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.028+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.028+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01a800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.040+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.040+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.052+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.052+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ff580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.056+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.056+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5daa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.064+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.064+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fd480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.072+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.072+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fdf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.080+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.080+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c046600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.084+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.084+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ffc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.084+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.084+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.088+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.088+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.092+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.092+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.100+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.100 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.100+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.104+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.105 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.104+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.112+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.111 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.112+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.123 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.124+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.123 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.124+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1dab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.132+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:14:00.132+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1dad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:14:02.266 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:14:02.270 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:14:02.271 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:14:02.283 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:14:03.682 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:14:03.683 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:14:03.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438843 2023-09-23T03:14:03.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438843 -le 1695440506 ]] 2023-09-23T03:14:03.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:14:03.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:14:03.687 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:14:13.723 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:14:13.732 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:14:13.739 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:14:13.748 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:14:15.053 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:14:15.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:14:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438855 2023-09-23T03:14:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438855 -le 1695440506 ]] 2023-09-23T03:14:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:14:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:14:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:14:25.059 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:14:25.060 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:14:25.060 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:14:25.063 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:14:26.189 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:14:26.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:14:26.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438866 2023-09-23T03:14:26.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438866 -le 1695440506 ]] 2023-09-23T03:14:26.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:14:26.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:14:26.191 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:14:36.191 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:14:36.192 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:14:36.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:14:36.207 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:14:37.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3586892185 2023-09-23T03:14:37.450 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:14:37.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438877 2023-09-23T03:14:37.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438877 -le 1695440506 ]] 2023-09-23T03:14:37.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3586892185 ]] 2023-09-23T03:14:37.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3586892185 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:14:37.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:14:46.972 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.971+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:14:46.972 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:14:46.972 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:14:46.974 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:46.975+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:14:47.463 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:14:47.464 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:14:47.467 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:14:47.482 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:14:48.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:14:48.765 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:14:48.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438888 2023-09-23T03:14:48.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438888 -le 1695440506 ]] 2023-09-23T03:14:48.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:14:48.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\5\8\6\8\9\2\1\8\5 ]] 2023-09-23T03:14:48.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/3739782250 2023-09-23T03:14:49.500 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.503+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:14:49.503 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.507+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:14:49.509 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.511+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:14:49.509 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.511+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:14:49.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.511+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:14:49.515 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.515+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:14:49.515 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.515+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:14:49.519 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.519+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:14:49.519 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.519+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:14:49.519 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.519+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:14:49.519 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:14:49.519+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:14:50.675 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/3739782250 until 2023-09-23T04:14:49.924712+0000 (3600 sec) 2023-09-23T03:14:50.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:14:50.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/3739782250 2023-09-23T03:14:51.664 INFO:tasks.workunit.client.0.smithi002.stderr:listed 29 entries 2023-09-23T03:14:51.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/3739782250 2023-09-23T03:14:51.709 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.010+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.010+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16addde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aaf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b1fd500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.018+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.022+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16bee5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.021 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.022+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.021 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.022+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c01ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.026+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.026+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c01bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.030+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.030+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c01be00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.033 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.034+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.033 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.034+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a1080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.038+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.038+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1c3180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.042+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.042+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ff600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.046+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.046+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3fe280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.050+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.050+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1daf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.062+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.062+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.062+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1db180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.062+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1db380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.066+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.066+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1db480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.070+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.070+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1db680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.074+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.074+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1db880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.078+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.078+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1dba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.082+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:00.082+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c1dbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:15:01.719 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:01.719 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:01.720 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:01.730 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:02.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:15:02.987 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:02.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438902 2023-09-23T03:15:02.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438902 -le 1695440506 ]] 2023-09-23T03:15:02.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:15:02.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:02.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:13.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:13.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:13.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:13.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:14.284 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:15:14.284 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:14.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438914 2023-09-23T03:15:14.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438914 -le 1695440506 ]] 2023-09-23T03:15:14.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:15:14.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:14.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:24.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:24.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:24.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:24.334 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:25.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:15:25.400 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:25.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438925 2023-09-23T03:15:25.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438925 -le 1695440506 ]] 2023-09-23T03:15:25.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:15:25.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:25.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:35.409 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:35.414 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:35.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:35.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:36.488 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:15:36.488 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:36.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438936 2023-09-23T03:15:36.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438936 -le 1695440506 ]] 2023-09-23T03:15:36.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:15:36.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:36.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:46.502 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:46.502 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:46.502 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:46.507 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:47.676 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/3739782250 2023-09-23T03:15:47.684 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:47.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438947 2023-09-23T03:15:47.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438947 -le 1695440506 ]] 2023-09-23T03:15:47.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/3739782250 ]] 2023-09-23T03:15:47.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/3739782250 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:47.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:15:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:15:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:15:57.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:15:57.697 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:15:58.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:15:58.867 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:15:58.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438958 2023-09-23T03:15:58.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438958 -le 1695440506 ]] 2023-09-23T03:15:58.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:15:58.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\3\7\3\9\7\8\2\2\5\0 ]] 2023-09-23T03:15:58.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2352963285 2023-09-23T03:15:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:15:59.996+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.000+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bef1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.000+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.001 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fdd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.008+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3fe980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.013 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.012+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.014 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.012+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5daa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.016+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.016+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1dbf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.027 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.028+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.028+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1dbc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.032+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.032 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.032+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.036+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.036+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.040+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.040+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1dbd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.044+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.044+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3fe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.052+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.052+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee4f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.052+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.052+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.056+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.056+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.060+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.060+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.068+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.068+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.072+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.071 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.072+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.076+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.076+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.080+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.080 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:16:00.080+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:16:00.711 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2352963285 until 2023-09-23T04:15:59.996658+0000 (3600 sec) 2023-09-23T03:16:00.744 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:16:00.744 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2352963285 2023-09-23T03:16:01.923 INFO:tasks.workunit.client.0.smithi002.stderr:listed 30 entries 2023-09-23T03:16:02.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2352963285 2023-09-23T03:16:02.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:16:12.018 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:16:12.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:16:12.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:16:12.027 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:16:13.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:16:13.287 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:16:13.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438973 2023-09-23T03:16:13.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438973 -le 1695440506 ]] 2023-09-23T03:16:13.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:16:13.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:16:13.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:16:23.305 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:16:23.305 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:16:23.305 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:16:23.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:16:24.628 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:16:24.628 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:16:24.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438984 2023-09-23T03:16:24.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438984 -le 1695440506 ]] 2023-09-23T03:16:24.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:16:24.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:16:24.629 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:16:30.080 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.072+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:16:30.087 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.080+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:16:30.087 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.080+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:16:30.088 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.084+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:16:30.088 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.084+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:16:30.095 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:16:30.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:16:30.092+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:16:34.641 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:16:34.645 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:16:34.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:16:34.655 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:16:35.813 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:16:35.816 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:16:35.819 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695438995 2023-09-23T03:16:35.819 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695438995 -le 1695440506 ]] 2023-09-23T03:16:35.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:16:35.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:16:35.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:16:45.825 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:16:45.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:16:45.834 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:16:45.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:16:46.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:16:46.967 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:16:46.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439006 2023-09-23T03:16:46.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439006 -le 1695440506 ]] 2023-09-23T03:16:46.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:16:46.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:16:46.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:16:56.970 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:16:56.972 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:16:56.975 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:16:56.979 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:16:58.385 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2352963285 2023-09-23T03:16:58.386 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:16:58.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439018 2023-09-23T03:16:58.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439018 -le 1695440506 ]] 2023-09-23T03:16:58.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2352963285 ]] 2023-09-23T03:16:58.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2352963285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:16:58.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:02.206 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:02.206 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:02.206 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:02.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:02.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:02.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:02.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:02.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:02.203+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:04.554 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.555+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:04.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.563+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:17:04.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.563+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:04.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.563+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:04.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.563+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:04.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.567+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:04.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.567+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:04.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.567+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:04.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.567+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:04.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.847+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:17:04.851 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.851+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:04.852 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:17:04.852 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:04.853 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:17:04.853 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:17:04.853 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:04.853 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:04.853 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:17:04.855 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:04.855 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:04.855 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:04.855 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:04.855+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:08.405 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:17:08.406 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:17:08.407 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:17:08.425 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:17:08.777 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.775+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:08.788 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:08.787+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:09.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:17:09.526 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:17:09.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439029 2023-09-23T03:17:09.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439029 -le 1695440506 ]] 2023-09-23T03:17:09.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:17:09.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\3\5\2\9\6\3\2\8\5 ]] 2023-09-23T03:17:09.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1944658510 2023-09-23T03:17:10.789 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:10.790+0000 7f3994226640 -1 librbd::ImageWatcher: 0x55c16c283800 image watch failed: 94289217983488, (107) Transport endpoint is not connected 2023-09-23T03:17:10.789 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:10.790+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c283800 handle_error: handle=94289217983488: (107) Transport endpoint is not connected 2023-09-23T03:17:10.789 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:10.790+0000 7f3994226640 -1 librbd::watcher::RewatchRequest: 0x55c16bab9090 handle_unwatch client blocklisted 2023-09-23T03:17:10.790 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:10.790+0000 7f3994226640 -1 librbd::Watcher: 0x55c16c283800 handle_rewatch: client blocklisted 2023-09-23T03:17:11.791 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1944658510 until 2023-09-23T04:17:11.042793+0000 (3600 sec) 2023-09-23T03:17:11.801 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994226640 -1 librbd::SnapshotCreateRequest: failed to notify unquiesce: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994226640 -1 librbd::mirror::snapshot::CreatePrimaryRequest: 0x55c16aef6a90 handle_create_snapshot: failed to create mirror snapshot: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994226640 -1 librbd::io::AioCompletion: 0x55c16c3c66e0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994a27640 -1 librbd::image::RefreshRequest: failed to retrieve mutable metadata: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994a27640 -1 librbd::io::AioCompletion: 0x55c16c7f82c0 fail: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994a27640 -1 librbd::exclusive_lock::PreReleaseRequest: 0x55c16c12de40 handle_set_require_lock: failed to set lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.802 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.802+0000 7f3994a27640 -1 librbd::object_map::UnlockRequest: failed to release object map lock: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.806 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:17:11.806+0000 7f3994a27640 -1 librbd::managed_lock::ReleaseRequest: 0x55c16bf1f200 handle_unlock: failed to unlock: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:17:11.835 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1944658510 2023-09-23T03:17:11.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:17:12.671 INFO:tasks.workunit.client.0.smithi002.stderr:listed 31 entries 2023-09-23T03:17:12.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1944658510 2023-09-23T03:17:12.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:16.575 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.566+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:17:16.576 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.566+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:17:16.576 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.574+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:16.580 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:17:16.581 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:16.581 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:16.583 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:16.583 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:16.583 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:16.583 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:16.582+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:22.708 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:17:22.718 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:17:22.720 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:17:22.723 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:17:23.935 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:17:23.935 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:17:23.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439043 2023-09-23T03:17:23.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439043 -le 1695440506 ]] 2023-09-23T03:17:23.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:17:23.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:17:23.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:33.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:17:33.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:17:33.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:17:33.960 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:17:35.060 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:17:35.060 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:17:35.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439055 2023-09-23T03:17:35.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439055 -le 1695440506 ]] 2023-09-23T03:17:35.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:17:35.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:17:35.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:39.975 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.970+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:39.976 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.974+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:17:39.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.974+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:39.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:39.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:39.988 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.986+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:39.988 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.986+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:39.988 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.986+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:39.988 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:39.986+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:17:45.067 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:17:45.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:17:45.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:17:45.079 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:17:46.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:17:46.151 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:17:46.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439066 2023-09-23T03:17:46.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439066 -le 1695440506 ]] 2023-09-23T03:17:46.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:17:46.152 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:17:46.153 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:56.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:17:56.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:17:56.158 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:17:56.176 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:17:57.326 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:17:57.337 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:17:57.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439077 2023-09-23T03:17:57.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439077 -le 1695440506 ]] 2023-09-23T03:17:57.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:17:57.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:17:57.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:17:58.835 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.833+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:17:58.843 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:17:58.843 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:17:58.844 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:17:58.845+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:18:07.354 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:18:07.354 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:18:07.354 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:18:07.355 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:18:08.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1944658510 2023-09-23T03:18:08.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:18:08.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439088 2023-09-23T03:18:08.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439088 -le 1695440506 ]] 2023-09-23T03:18:08.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1944658510 ]] 2023-09-23T03:18:08.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1944658510 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:18:08.478 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:18:18.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:18:18.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:18:18.496 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:18:18.496 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:18:19.446 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:19.445+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:18:20.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:18:20.085 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:18:20.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439100 2023-09-23T03:18:20.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439100 -le 1695440506 ]] 2023-09-23T03:18:20.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:18:20.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\9\4\4\6\5\8\5\1\0 ]] 2023-09-23T03:18:20.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/4113803064 2023-09-23T03:18:22.761 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/4113803064 until 2023-09-23T04:18:22.088965+0000 (3600 sec) 2023-09-23T03:18:22.800 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:18:22.800 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/4113803064 2023-09-23T03:18:23.738 INFO:tasks.workunit.client.0.smithi002.stderr:listed 32 entries 2023-09-23T03:18:23.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/4113803064 2023-09-23T03:18:23.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:18:24.127 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.125+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:18:24.128 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.129+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:18:24.135 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.133+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:18:24.135 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.133+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:18:24.138 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.137+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:18:24.138 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.137+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:18:24.138 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.137+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:18:24.142 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.141+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:18:24.154 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.149+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:18:24.154 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.149+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:18:24.154 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.149+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:18:24.154 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:24.149+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:18:25.357 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:18:25.357 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:18:25.357 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:18:25.357 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:18:25.359 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:18:25.359 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:18:25.359 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:18:25.359 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:25.357+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:18:27.768 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.761+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:18:27.768 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.761+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:18:27.776 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:18:27.777 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:18:27.777 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:18:27.777 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:18:27.777 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:18:27.769+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:18:33.785 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:18:33.786 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:18:33.788 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:18:33.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:18:35.012 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:18:35.014 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:18:35.015 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439115 2023-09-23T03:18:35.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439115 -le 1695440506 ]] 2023-09-23T03:18:35.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:18:35.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:18:35.016 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:18:45.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:18:45.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:18:45.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:18:45.026 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:18:46.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:18:46.067 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:18:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439126 2023-09-23T03:18:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439126 -le 1695440506 ]] 2023-09-23T03:18:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:18:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:18:46.070 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:18:56.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:18:56.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:18:56.095 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:18:56.105 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:18:57.330 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:18:57.332 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:18:57.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439137 2023-09-23T03:18:57.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439137 -le 1695440506 ]] 2023-09-23T03:18:57.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:18:57.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:18:57.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:19:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.000+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.003 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.000+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3aad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.004+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.004+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b5cbb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.008+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.008+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1daf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.016+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.016 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.016+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c01b080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.024 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.024+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.025 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.024+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1db680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.040+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.040+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.056+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.056+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e7e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.060+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.060+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.068+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.068+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.090 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.090 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.121 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.120+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.121 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.120+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.132+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.132+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.132+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.132+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.136 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.136+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.136 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.136+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.139 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.136+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.139 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.140+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.147 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.144+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.147 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.144+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c255080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.164+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.164+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.164+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c255a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:00.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:19:00.164+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c255880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:19:07.340 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:19:07.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:19:07.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:19:07.376 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:19:08.851 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:19:08.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:19:08.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439148 2023-09-23T03:19:08.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439148 -le 1695440506 ]] 2023-09-23T03:19:08.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:19:08.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:19:08.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:19:18.856 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:19:18.857 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:19:18.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:19:18.872 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:19:20.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4113803064 2023-09-23T03:19:20.178 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:19:20.178 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439160 2023-09-23T03:19:20.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439160 -le 1695440506 ]] 2023-09-23T03:19:20.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4113803064 ]] 2023-09-23T03:19:20.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4113803064 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:19:20.179 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:19:30.189 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:19:30.190 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:19:30.200 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:19:30.224 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:19:31.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:19:31.475 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:19:31.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439171 2023-09-23T03:19:31.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439171 -le 1695440506 ]] 2023-09-23T03:19:31.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:19:31.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\1\3\8\0\3\0\6\4 ]] 2023-09-23T03:19:31.476 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/4188811661 2023-09-23T03:19:34.092 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/4188811661 until 2023-09-23T04:19:33.101029+0000 (3600 sec) 2023-09-23T03:19:34.146 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:19:34.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/4188811661 2023-09-23T03:19:35.132 INFO:tasks.workunit.client.0.smithi002.stderr:listed 33 entries 2023-09-23T03:19:35.199 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/4188811661 2023-09-23T03:19:35.199 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:19:45.205 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:19:45.205 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:19:45.206 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:19:45.215 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:19:46.583 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:19:46.583 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:19:46.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439186 2023-09-23T03:19:46.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439186 -le 1695440506 ]] 2023-09-23T03:19:46.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:19:46.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:19:46.584 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:19:48.963 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.963+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:19:48.964 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.963+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:19:48.970 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:19:48.971 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:19:48.971+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:19:56.594 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:19:56.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:19:56.601 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:19:56.626 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:19:57.741 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:19:57.741 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:19:57.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439197 2023-09-23T03:19:57.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439197 -le 1695440506 ]] 2023-09-23T03:19:57.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:19:57.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:19:57.747 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:20:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.002+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.002+0000 7f3994226640 -1 librbd::ImageState: 0x55c16be92800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.006+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.006+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.038+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.038 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.038+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.044 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.054+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.054+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.056 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.059 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.066+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.066+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.078+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.078+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.086+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.086+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.088 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.090+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.090+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.090+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.091 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.090+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7faa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.094+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.094+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.098+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.098+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.106+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.106+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.113 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.114+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.113 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.114+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.118+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.119 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.118+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fbb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.129 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.130+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:00.130 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:20:00.130+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:20:07.749 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:20:07.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:20:07.753 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:20:07.757 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:20:08.933 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:20:08.935 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:20:08.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439208 2023-09-23T03:20:08.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439208 -le 1695440506 ]] 2023-09-23T03:20:08.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:20:08.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:20:08.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:20:18.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:20:18.951 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:20:18.952 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:20:18.959 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:20:20.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:20:20.136 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:20:20.136 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439220 2023-09-23T03:20:20.136 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439220 -le 1695440506 ]] 2023-09-23T03:20:20.136 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:20:20.136 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:20:20.137 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:20:20.691 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:20:20.691 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:20:20.692 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:20:20.690+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:20:30.138 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:20:30.148 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:20:30.150 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:20:30.155 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:20:31.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4188811661 2023-09-23T03:20:31.248 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:20:31.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439231 2023-09-23T03:20:31.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439231 -le 1695440506 ]] 2023-09-23T03:20:31.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4188811661 ]] 2023-09-23T03:20:31.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4188811661 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:20:31.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:20:41.251 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:20:41.259 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:20:41.282 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:20:41.289 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:20:42.596 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2891944060 2023-09-23T03:20:42.597 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:20:42.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439242 2023-09-23T03:20:42.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439242 -le 1695440506 ]] 2023-09-23T03:20:42.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2891944060 ]] 2023-09-23T03:20:42.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2891944060 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\1\8\8\8\1\1\6\6\1 ]] 2023-09-23T03:20:42.602 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2891944060 2023-09-23T03:20:45.084 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2891944060 until 2023-09-23T04:20:44.416620+0000 (3600 sec) 2023-09-23T03:20:45.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:20:45.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2891944060 2023-09-23T03:20:46.252 INFO:tasks.workunit.client.0.smithi002.stderr:listed 34 entries 2023-09-23T03:20:46.306 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2891944060 2023-09-23T03:20:46.306 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:20:56.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:20:56.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:20:56.314 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:20:56.329 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:20:57.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2891944060 2023-09-23T03:20:57.538 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:20:57.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439257 2023-09-23T03:20:57.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439257 -le 1695440506 ]] 2023-09-23T03:20:57.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2891944060 ]] 2023-09-23T03:20:57.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2891944060 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\9\1\9\4\4\0\6\0 ]] 2023-09-23T03:20:57.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.009+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.010 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.009+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.013+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.013+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fdf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.017+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.017+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1db880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.021+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.021+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.037+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.040 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.037+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c074480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.049+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.050 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.049+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.053+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.053+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.057+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.057 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.057+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.061+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.061 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.061+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1db880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.073+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.073+0000 7f3993a25640 -1 librbd::ImageState: 0x55c1669e6780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.077+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.077+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fdf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.081+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.081+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1a0500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.085+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.085+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.097+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.097+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ab680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.105+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.106 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.105+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1da100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.109+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.109+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c7fb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.117 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.117+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.117 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.117+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c7fb380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.121+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.121+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.125+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.125+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c255200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.133 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.133+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:00.133 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:00.133+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c254380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:21:07.540 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:21:07.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:21:07.551 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:21:07.554 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:21:08.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2891944060 2023-09-23T03:21:08.801 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:21:08.802 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439268 2023-09-23T03:21:08.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439268 -le 1695440506 ]] 2023-09-23T03:21:08.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2891944060 ]] 2023-09-23T03:21:08.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2891944060 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\9\1\9\4\4\0\6\0 ]] 2023-09-23T03:21:08.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:18.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:21:18.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:21:18.828 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:21:18.837 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:21:20.158 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2891944060 2023-09-23T03:21:20.159 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:21:20.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439280 2023-09-23T03:21:20.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439280 -le 1695440506 ]] 2023-09-23T03:21:20.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2891944060 ]] 2023-09-23T03:21:20.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2891944060 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\9\1\9\4\4\0\6\0 ]] 2023-09-23T03:21:20.161 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:30.172 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:21:30.177 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:21:30.187 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:21:30.188 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:21:31.333 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2891944060 2023-09-23T03:21:31.333 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:21:31.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439291 2023-09-23T03:21:31.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439291 -le 1695440506 ]] 2023-09-23T03:21:31.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2891944060 ]] 2023-09-23T03:21:31.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2891944060 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\9\1\9\4\4\0\6\0 ]] 2023-09-23T03:21:31.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:41.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:21:41.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:21:41.337 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:21:41.377 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:21:42.588 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:21:42.588 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:21:42.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439302 2023-09-23T03:21:42.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439302 -le 1695440506 ]] 2023-09-23T03:21:42.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:21:42.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\8\9\1\9\4\4\0\6\0 ]] 2023-09-23T03:21:42.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2146552285 2023-09-23T03:21:45.194 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2146552285 until 2023-09-23T04:21:44.568276+0000 (3600 sec) 2023-09-23T03:21:45.254 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:21:45.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2146552285 2023-09-23T03:21:46.527 INFO:tasks.workunit.client.0.smithi002.stderr:listed 35 entries 2023-09-23T03:21:46.562 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2146552285 2023-09-23T03:21:46.562 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:56.567 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:21:56.567 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:21:56.567 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:21:56.568 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:21:57.803 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:21:57.815 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:21:57.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439317 2023-09-23T03:21:57.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439317 -le 1695440506 ]] 2023-09-23T03:21:57.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:21:57.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:21:57.818 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:21:59.999 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:59.996+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:21:59.996+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16aef3580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.000+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.002 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.000+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16b4a1500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.004+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.005 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.004+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3aad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.012+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.012+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c255a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.020+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.020+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.020+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c3ab680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.020 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.020+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c7faa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c7fae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994a27640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994a27640 -1 librbd::ImageState: 0x55c16c7fbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.028+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.032+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.035 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.032+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.048+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.051 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.048+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.052+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.052+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1da100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.060+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.062 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.060+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fbf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.068+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.068+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c4d8100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.072+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.072+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c4d8300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.076+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.076+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c4d8500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.092+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.092+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c4d8700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.096+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.096+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c4d8900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.100+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.101 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.100+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fbc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.104+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:00.104 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:22:00.104+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:22:07.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:22:07.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:22:07.826 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:22:07.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:22:08.925 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:22:08.925 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:22:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439328 2023-09-23T03:22:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439328 -le 1695440506 ]] 2023-09-23T03:22:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:22:08.926 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:22:08.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:22:18.928 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:22:18.931 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:22:18.939 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:22:18.947 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:22:20.146 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:22:20.146 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:22:20.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439340 2023-09-23T03:22:20.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439340 -le 1695440506 ]] 2023-09-23T03:22:20.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:22:20.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:22:20.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:22:30.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:22:30.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:22:30.149 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:22:30.159 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:22:31.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:22:31.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:22:31.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439351 2023-09-23T03:22:31.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439351 -le 1695440506 ]] 2023-09-23T03:22:31.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:22:31.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:22:31.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:22:37.494 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:22:37.491+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:22:37.495 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:22:37.491+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:22:37.495 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:22:37.491+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:22:41.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:22:41.373 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:22:41.390 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:22:41.394 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:22:42.511 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2146552285 2023-09-23T03:22:42.512 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:22:42.512 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439362 2023-09-23T03:22:42.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439362 -le 1695440506 ]] 2023-09-23T03:22:42.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2146552285 ]] 2023-09-23T03:22:42.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2146552285 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:22:42.513 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:22:52.515 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:22:52.522 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:22:52.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:22:52.536 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:22:53.804 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:22:53.804 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:22:53.805 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439373 2023-09-23T03:22:53.805 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439373 -le 1695440506 ]] 2023-09-23T03:22:53.805 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:22:53.805 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\1\4\6\5\5\2\2\8\5 ]] 2023-09-23T03:22:53.805 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/166270078 2023-09-23T03:22:56.211 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/166270078 until 2023-09-23T04:22:55.538386+0000 (3600 sec) 2023-09-23T03:22:56.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:22:56.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/166270078 2023-09-23T03:22:57.278 INFO:tasks.workunit.client.0.smithi002.stderr:listed 36 entries 2023-09-23T03:22:57.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/166270078 2023-09-23T03:22:57.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:23:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.003+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.000 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.003+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3abe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.007+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.007+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c253680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.007+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.006 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.007+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.011+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.009 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.011+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.015+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.015+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.019+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.019+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.023+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.022 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.023+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fc580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.047+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.051+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.059+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.059+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ab080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.067+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.067+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3abe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.079+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.076 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.079+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1c3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.095+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.092 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bbab400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.095+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.095 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.103+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.102 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.103+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.111+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.108 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.111+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.115+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.112 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.115+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1da480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.116 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.119+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.116 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.119+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1da100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.131+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.128 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.131+0000 7f3994226640 -1 librbd::ImageState: 0x55c16b1fd480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.135+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.132 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.135+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.136 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.139+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:00.136 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:23:00.139+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:23:07.320 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:23:07.320 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:23:07.320 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:23:07.323 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:23:08.591 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:23:08.591 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:23:08.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439388 2023-09-23T03:23:08.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439388 -le 1695440506 ]] 2023-09-23T03:23:08.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:23:08.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:23:08.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:23:18.594 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:23:18.596 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:23:18.597 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:23:18.602 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:23:19.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:23:19.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:23:19.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439399 2023-09-23T03:23:19.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439399 -le 1695440506 ]] 2023-09-23T03:23:19.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:23:19.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:23:19.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:23:28.753 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.754+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:23:28.769 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:23:28.770 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:23:28.771 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:28.770+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:23:29.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:23:29.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:23:29.874 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:23:29.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:23:31.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:23:31.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:23:31.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439411 2023-09-23T03:23:31.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439411 -le 1695440506 ]] 2023-09-23T03:23:31.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:23:31.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:23:31.022 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:23:41.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:23:41.027 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:23:41.029 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:23:41.046 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:23:42.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:23:42.335 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:23:42.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439422 2023-09-23T03:23:42.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439422 -le 1695440506 ]] 2023-09-23T03:23:42.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:23:42.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:23:42.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:23:44.394 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:44.394+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:23:44.420 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:44.422+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:23:44.420 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:44.422+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:23:44.420 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:44.422+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:23:44.420 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:23:44.422+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:23:52.343 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:23:52.343 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:23:52.343 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:23:52.354 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:23:53.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/166270078 2023-09-23T03:23:53.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:23:53.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439433 2023-09-23T03:23:53.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439433 -le 1695440506 ]] 2023-09-23T03:23:53.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/166270078 ]] 2023-09-23T03:23:53.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/166270078 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:23:53.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:24:03.532 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:24:03.533 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:24:03.533 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:24:03.537 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:24:04.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:24:04.857 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:24:04.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439444 2023-09-23T03:24:04.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439444 -le 1695440506 ]] 2023-09-23T03:24:04.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:24:04.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\6\2\7\0\0\7\8 ]] 2023-09-23T03:24:04.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/1654913027 2023-09-23T03:24:07.225 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/1654913027 until 2023-09-23T04:24:06.562699+0000 (3600 sec) 2023-09-23T03:24:07.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:24:07.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/1654913027 2023-09-23T03:24:08.130 INFO:tasks.workunit.client.0.smithi002.stderr:listed 37 entries 2023-09-23T03:24:08.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/1654913027 2023-09-23T03:24:08.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:24:18.167 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:24:18.168 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:24:18.169 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:24:18.183 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:24:19.311 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:24:19.313 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:24:19.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439459 2023-09-23T03:24:19.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439459 -le 1695440506 ]] 2023-09-23T03:24:19.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:24:19.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:24:19.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:24:29.318 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:24:29.318 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:24:29.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:24:29.353 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:24:29.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.569+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:24:29.587 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.581+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:24:29.588 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.585+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:24:29.588 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.585+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:24:29.588 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.585+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:24:29.588 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:24:29.585+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:24:30.590 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:24:30.591 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:24:30.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439470 2023-09-23T03:24:30.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439470 -le 1695440506 ]] 2023-09-23T03:24:30.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:24:30.592 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:24:30.593 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:24:40.593 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:24:40.594 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:24:40.594 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:24:40.623 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:24:41.597 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:24:41.598 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:24:41.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439481 2023-09-23T03:24:41.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439481 -le 1695440506 ]] 2023-09-23T03:24:41.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:24:41.599 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:24:41.599 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:24:51.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:24:51.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:24:51.611 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:24:51.620 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:24:52.837 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:24:52.838 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:24:52.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439492 2023-09-23T03:24:52.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439492 -le 1695440506 ]] 2023-09-23T03:24:52.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:24:52.838 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:24:52.839 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:25:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.000+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.000+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.016+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.017 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.016+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.016+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.018 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.016+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.023 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.020+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.023 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.020+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.028+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.028 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.028+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.052+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.054 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.052+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.056+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.058 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.056+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.064+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.066 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.064+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.072+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.072 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.072+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.072+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.075 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.072+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.084+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.085 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.084+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3aab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.088+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.089 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.088+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.108+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.109 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.108+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.116+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.116+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.116+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.116+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.120+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.122 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.120+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.124+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.127 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.124+0000 7f3994226640 -1 librbd::ImageState: 0x55c16bee4600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.128+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.131 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.128+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.132+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:00.135 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:25:00.132+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:25:02.848 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:25:02.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:25:02.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:25:02.856 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:25:04.150 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/1654913027 2023-09-23T03:25:04.150 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:25:04.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439504 2023-09-23T03:25:04.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439504 -le 1695440506 ]] 2023-09-23T03:25:04.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/1654913027 ]] 2023-09-23T03:25:04.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/1654913027 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:25:04.151 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:25:14.153 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:25:14.153 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:25:14.154 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:25:14.178 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:25:15.540 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:25:15.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:25:15.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439515 2023-09-23T03:25:15.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439515 -le 1695440506 ]] 2023-09-23T03:25:15.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:25:15.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\1\6\5\4\9\1\3\0\2\7 ]] 2023-09-23T03:25:15.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/2531452558 2023-09-23T03:25:18.249 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/2531452558 until 2023-09-23T04:25:17.461019+0000 (3600 sec) 2023-09-23T03:25:18.276 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:25:18.276 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/2531452558 2023-09-23T03:25:19.226 INFO:tasks.workunit.client.0.smithi002.stderr:listed 38 entries 2023-09-23T03:25:19.281 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/2531452558 2023-09-23T03:25:19.281 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:25:28.511 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:25:28.511 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:25:28.511 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:25:28.511 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:25:28.511 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:25:28.512 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:25:28.508+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:25:29.284 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:25:29.284 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:25:29.284 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:25:29.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:25:30.614 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:25:30.615 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:25:30.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439530 2023-09-23T03:25:30.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439530 -le 1695440506 ]] 2023-09-23T03:25:30.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:25:30.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:25:30.616 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:25:40.624 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:25:40.627 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:25:40.630 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:25:40.638 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:25:41.933 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:25:41.933 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:25:41.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439541 2023-09-23T03:25:41.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439541 -le 1695440506 ]] 2023-09-23T03:25:41.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:25:41.935 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:25:41.935 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:25:51.936 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:25:51.937 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:25:51.950 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:25:51.967 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:25:53.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:25:53.059 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:25:53.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439553 2023-09-23T03:25:53.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439553 -le 1695440506 ]] 2023-09-23T03:25:53.061 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:25:53.062 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:25:53.062 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:26:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.003+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.003+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.003+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c1c2a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.004 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.003+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ffa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.011 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3fff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.012 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ffc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.015 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.011+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16aeef700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.015+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.019 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.015+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16b1fca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.027+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.029 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.027+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16be92080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.031+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.031+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16bee5400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.031+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.031+0000 7f3993a25640 -1 librbd::ImageState: 0x55c16c3ab080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.043+0000 7f3993a25640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.045 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.043+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ab480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.047+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.049 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.047+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1da500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.059+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.059+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c1dad00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.063+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.064 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.063+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.067+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.068 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.067+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c074f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.071+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.073 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.071+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ab000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.075+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.077 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.075+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.079+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.082 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.079+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.083+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.086 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.083+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c254900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.091+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.094 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.091+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.095+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:00.098 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:26:00.095+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:26:03.069 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:26:03.073 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:26:03.076 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:26:03.085 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:26:04.503 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:26:04.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:26:04.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439564 2023-09-23T03:26:04.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439564 -le 1695440506 ]] 2023-09-23T03:26:04.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:26:04.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:26:04.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:26:14.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:26:14.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:26:14.542 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:26:14.544 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:26:16.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/2531452558 2023-09-23T03:26:16.047 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:26:16.048 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439576 2023-09-23T03:26:16.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439576 -le 1695440506 ]] 2023-09-23T03:26:16.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/2531452558 ]] 2023-09-23T03:26:16.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/2531452558 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:26:16.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:26:26.051 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:26:26.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:26:26.054 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:26:26.060 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:26:27.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:26:27.300 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:26:27.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439587 2023-09-23T03:26:27.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439587 -le 1695440506 ]] 2023-09-23T03:26:27.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:26:27.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\2\5\3\1\4\5\2\5\5\8 ]] 2023-09-23T03:26:27.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist add 172.21.15.2:0/4250006422 2023-09-23T03:26:29.903 INFO:tasks.workunit.client.0.smithi002.stderr:blocklisting 172.21.15.2:0/4250006422 until 2023-09-23T04:26:29.251931+0000 (3600 sec) 2023-09-23T03:26:29.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep -q 172.21.15.2:0/4250006422 2023-09-23T03:26:29.973 INFO:tasks.workunit.client.0.smithi002.stderr:+ ceph osd blocklist ls 2023-09-23T03:26:31.084 INFO:tasks.workunit.client.0.smithi002.stderr:listed 39 entries 2023-09-23T03:26:31.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ PREV_CLIENT=172.21.15.2:0/4250006422 2023-09-23T03:26:31.129 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:26:34.423 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:34.415+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:26:34.448 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:34.447+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:26:34.448 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:34.447+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:26:37.623 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.619+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:26:37.628 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.627+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:26:37.628 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.627+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:26:37.628 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.631+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:26:37.630 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.631+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:26:37.636 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:26:37.636 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:26:37.636 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:26:37.636 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:26:37.637 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:26:37.639+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:26:41.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:26:41.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:26:41.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:26:41.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:26:42.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:26:42.363 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:26:42.364 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439602 2023-09-23T03:26:42.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439602 -le 1695440506 ]] 2023-09-23T03:26:42.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:26:42.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:26:42.365 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:26:52.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:26:52.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:26:52.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:26:52.403 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:26:53.462 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:26:53.463 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:26:53.463 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439613 2023-09-23T03:26:53.463 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439613 -le 1695440506 ]] 2023-09-23T03:26:53.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:26:53.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:26:53.464 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:00.007 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3ff500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.010+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c3aa900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.036 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.038+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.038+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c255c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.038+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.037 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.038+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.041 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.042+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.042 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.042+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.050+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.048 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.050+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fa780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.054+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.052 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.054+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.058+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.055 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.058+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.062+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.060 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.062+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7faa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.062+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.063 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.062+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.066+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.065 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.066+0000 7f3994226640 -1 librbd::ImageState: 0x55c1669e7500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.067 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.070+0000 7f3994226640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:00.069 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:27:00.070+0000 7f3994226640 -1 librbd::ImageState: 0x55c16c7fb680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-09-23T03:27:03.473 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:03.473 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:03.476 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:27:03.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:04.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:27:04.739 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:27:04.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439624 2023-09-23T03:27:04.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439624 -le 1695440506 ]] 2023-09-23T03:27:04.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:27:04.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:27:04.740 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:14.743 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:27:14.750 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:14.751 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:14.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:15.895 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:27:15.896 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:27:15.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439635 2023-09-23T03:27:15.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439635 -le 1695440506 ]] 2023-09-23T03:27:15.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:27:15.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:27:15.905 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:25.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:25.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:25.925 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:27:25.939 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:26.998 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:27:26.998 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:27:26.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439647 2023-09-23T03:27:26.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439647 -le 1695440506 ]] 2023-09-23T03:27:26.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:27:26.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:27:26.999 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:37.028 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:37.036 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:37.045 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:37.047 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:27:38.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:27:38.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:27:38.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439658 2023-09-23T03:27:38.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439658 -le 1695440506 ]] 2023-09-23T03:27:38.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:27:38.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:27:38.453 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:48.454 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:48.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:48.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:48.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:27:49.773 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:27:49.775 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:27:49.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439669 2023-09-23T03:27:49.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439669 -le 1695440506 ]] 2023-09-23T03:27:49.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:27:49.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:27:49.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:27:59.787 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:27:59.789 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:27:59.791 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:27:59.824 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:01.063 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:01.064 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:01.066 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439681 2023-09-23T03:28:01.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439681 -le 1695440506 ]] 2023-09-23T03:28:01.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:01.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:01.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:28:11.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:28:11.097 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:11.098 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:28:11.107 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:28:12.353 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:12.353 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:12.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439692 2023-09-23T03:28:12.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439692 -le 1695440506 ]] 2023-09-23T03:28:12.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:12.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:12.354 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:28:22.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:28:22.374 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:28:22.378 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:28:22.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:23.990 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:23.999 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:24.002 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439704 2023-09-23T03:28:24.002 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439704 -le 1695440506 ]] 2023-09-23T03:28:24.002 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:24.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:24.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:28:34.020 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:28:34.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:34.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:28:34.022 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:28:35.035 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:35.035 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:35.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439715 2023-09-23T03:28:35.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439715 -le 1695440506 ]] 2023-09-23T03:28:35.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:35.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:35.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:28:42.203 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:28:42.204+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:28:42.206 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:28:42.208+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:28:45.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:28:45.045 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:45.059 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:28:45.065 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:28:46.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:46.279 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:46.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439726 2023-09-23T03:28:46.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439726 -le 1695440506 ]] 2023-09-23T03:28:46.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:46.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:46.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:28:56.290 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:28:56.293 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:28:56.295 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:28:56.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:28:57.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439737 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439737 -le 1695440506 ]] 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:28:57.336 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:29:07.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:29:07.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:29:07.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:29:07.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:29:08.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:29:08.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:29:08.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439748 2023-09-23T03:29:08.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439748 -le 1695440506 ]] 2023-09-23T03:29:08.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:29:08.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:29:08.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:29:18.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:29:18.525 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:29:18.527 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:29:18.547 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:29:19.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:29:19.734 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:29:19.735 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439759 2023-09-23T03:29:19.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439759 -le 1695440506 ]] 2023-09-23T03:29:19.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:29:19.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:29:19.736 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:29:29.752 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:29:29.756 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:29:29.762 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:29:29.765 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:29:31.122 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:29:31.122 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:29:31.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439771 2023-09-23T03:29:31.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439771 -le 1695440506 ]] 2023-09-23T03:29:31.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:29:31.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:29:31.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:29:41.130 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:29:41.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:29:41.131 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:29:41.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:29:42.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:29:42.444 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:29:42.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439782 2023-09-23T03:29:42.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439782 -le 1695440506 ]] 2023-09-23T03:29:42.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:29:42.444 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:29:42.445 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:29:47.838 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:29:47.827+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:29:49.138 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:29:49.131+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:29:49.139 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:29:49.139+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:29:52.456 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:29:52.456 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:29:52.461 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:29:52.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:29:53.660 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:29:53.647+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:29:53.660 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:29:53.651+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:29:53.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:29:53.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:29:53.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439793 2023-09-23T03:29:53.850 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439793 -le 1695440506 ]] 2023-09-23T03:29:53.850 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:29:53.850 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:29:53.850 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:00.570 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.551+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:30:00.571 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.559+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:30:00.571 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.559+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:30:00.571 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.563+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:30:00.572 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.563+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:30:00.572 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.563+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:30:00.572 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:00.563+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:30:03.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:03.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:03.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:30:03.884 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:05.199 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:30:05.199 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:30:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439805 2023-09-23T03:30:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439805 -le 1695440506 ]] 2023-09-23T03:30:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:30:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:30:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:15.217 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:15.218 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:30:15.226 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:15.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:16.435 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:30:16.445 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:30:16.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439816 2023-09-23T03:30:16.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439816 -le 1695440506 ]] 2023-09-23T03:30:16.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:30:16.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:30:16.447 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:26.462 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:30:26.468 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:26.468 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:26.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:27.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:30:27.557 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:30:27.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439827 2023-09-23T03:30:27.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439827 -le 1695440506 ]] 2023-09-23T03:30:27.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:30:27.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:30:27.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:37.569 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:37.572 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:30:37.574 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:37.578 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:38.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:30:38.580 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:30:38.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439838 2023-09-23T03:30:38.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439838 -le 1695440506 ]] 2023-09-23T03:30:38.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:30:38.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:30:38.582 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:48.583 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:30:48.584 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:48.584 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:48.586 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:49.004 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.002+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:30:49.006 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.006+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:30:49.006 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.006+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:30:49.026 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.026+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:30:49.027 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.026+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:30:49.041 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:30:49.041 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:30:49.041 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:30:49.041 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:30:49.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:30:49.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:30:49.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:30:49.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:30:49.042 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:30:49.042+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:30:49.724 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:30:49.728 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:30:49.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439849 2023-09-23T03:30:49.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439849 -le 1695440506 ]] 2023-09-23T03:30:49.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:30:49.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:30:49.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:30:59.735 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:30:59.735 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:30:59.735 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:30:59.738 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:00.925 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:00.926 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:00.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439860 2023-09-23T03:31:00.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439860 -le 1695440506 ]] 2023-09-23T03:31:00.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:00.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:00.927 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:31:10.934 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:31:10.936 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:31:10.941 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:31:10.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:11.962 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:11.966 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:11.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439871 2023-09-23T03:31:11.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439871 -le 1695440506 ]] 2023-09-23T03:31:11.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:11.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:11.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:31:21.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:31:21.974 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:31:21.975 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:22.003 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:31:23.002 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:23.002 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:23.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439883 2023-09-23T03:31:23.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439883 -le 1695440506 ]] 2023-09-23T03:31:23.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:23.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:23.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:31:33.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:31:33.008 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:31:33.010 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:31:33.014 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:34.071 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:34.075 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:34.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439894 2023-09-23T03:31:34.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439894 -le 1695440506 ]] 2023-09-23T03:31:34.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:34.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:34.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:31:44.081 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:31:44.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:31:44.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:31:44.091 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:44.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:44.984 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:44.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439904 2023-09-23T03:31:44.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439904 -le 1695440506 ]] 2023-09-23T03:31:44.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:44.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:44.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:31:46.482 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.481+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:31:46.483 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:31:46.484 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:31:46.484 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:31:46.484 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:31:46.485+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:31:54.991 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:31:54.992 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:31:54.997 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:31:54.998 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:31:56.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:31:56.084 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:31:56.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439916 2023-09-23T03:31:56.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439916 -le 1695440506 ]] 2023-09-23T03:31:56.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:31:56.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:31:56.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:32:06.092 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:32:06.092 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:32:06.093 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:32:06.128 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:32:07.271 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:32:07.271 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:32:07.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439927 2023-09-23T03:32:07.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439927 -le 1695440506 ]] 2023-09-23T03:32:07.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:32:07.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:32:07.272 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:32:17.282 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:32:17.285 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:32:17.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:32:17.289 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:32:18.515 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:32:18.521 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:32:18.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439938 2023-09-23T03:32:18.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439938 -le 1695440506 ]] 2023-09-23T03:32:18.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:32:18.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:32:18.522 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:32:28.541 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:32:28.546 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:32:28.552 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:32:28.571 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:32:29.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:32:29.821 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:32:29.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439949 2023-09-23T03:32:29.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439949 -le 1695440506 ]] 2023-09-23T03:32:29.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:32:29.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:32:29.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:32:39.830 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:32:39.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:32:39.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:32:39.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:32:41.160 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:32:41.160 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:32:41.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439961 2023-09-23T03:32:41.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439961 -le 1695440506 ]] 2023-09-23T03:32:41.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:32:41.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:32:41.162 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:32:51.167 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:32:51.170 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:32:51.173 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:32:51.174 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:32:52.755 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:32:52.761 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:32:52.764 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439972 2023-09-23T03:32:52.764 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439972 -le 1695440506 ]] 2023-09-23T03:32:52.764 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:32:52.764 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:32:52.764 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:02.769 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:02.769 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:33:02.770 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:02.794 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:03.725 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:33:03.726 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:33:03.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439983 2023-09-23T03:33:03.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439983 -le 1695440506 ]] 2023-09-23T03:33:03.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:33:03.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:33:03.728 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:09.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.563+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:33:09.573 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:33:09.574 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:09.575+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:33:13.729 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:13.730 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:33:13.730 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:13.735 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:33:15.055 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:33:15.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695439995 2023-09-23T03:33:15.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695439995 -le 1695440506 ]] 2023-09-23T03:33:15.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:33:15.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:33:15.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:25.066 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:25.067 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:25.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:25.099 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:33:26.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:33:26.345 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:33:26.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440006 2023-09-23T03:33:26.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440006 -le 1695440506 ]] 2023-09-23T03:33:26.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:33:26.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:33:26.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:34.334+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:33:34.332 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:34.334+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:33:36.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:36.353 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:36.354 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:36.356 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:33:37.002 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.002+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:33:37.007 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.010+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:33:37.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:33:37.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:33:37.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:33:37.025 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:33:37.026+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:33:37.694 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:33:37.698 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:33:37.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440017 2023-09-23T03:33:37.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440017 -le 1695440506 ]] 2023-09-23T03:33:37.699 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:33:37.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:33:37.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:47.702 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:47.705 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:47.718 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:33:47.721 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:49.267 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:33:49.267 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:33:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440029 2023-09-23T03:33:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440029 -le 1695440506 ]] 2023-09-23T03:33:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:33:49.268 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:33:49.269 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:33:59.274 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:33:59.276 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:33:59.279 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:33:59.282 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:00.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:00.575 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:00.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440040 2023-09-23T03:34:00.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440040 -le 1695440506 ]] 2023-09-23T03:34:00.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:00.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:00.576 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:34:10.577 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:34:10.578 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:34:10.582 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:34:10.599 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:11.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:11.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:11.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440051 2023-09-23T03:34:11.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440051 -le 1695440506 ]] 2023-09-23T03:34:11.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:11.842 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:11.843 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:34:21.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:34:21.856 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:34:21.864 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:34:21.892 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:23.190 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:23.191 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:23.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440063 2023-09-23T03:34:23.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440063 -le 1695440506 ]] 2023-09-23T03:34:23.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:23.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:23.194 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:34:33.208 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:34:33.213 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:34:33.215 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:33.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:34:34.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:34.327 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:34.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440074 2023-09-23T03:34:34.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440074 -le 1695440506 ]] 2023-09-23T03:34:34.337 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:34.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:34.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:34:43.641 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:34:43.642 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.637+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:34:43.868 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:34:43.865+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:34:44.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:34:44.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:34:44.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:34:44.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:45.431 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:45.432 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:45.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440085 2023-09-23T03:34:45.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440085 -le 1695440506 ]] 2023-09-23T03:34:45.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:45.432 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:45.433 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:34:55.437 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:34:55.447 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:34:55.449 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:34:55.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:34:56.902 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:34:56.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:34:56.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440096 2023-09-23T03:34:56.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440096 -le 1695440506 ]] 2023-09-23T03:34:56.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:34:56.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:34:56.907 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:06.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:35:06.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:35:06.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:35:06.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:35:08.480 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:35:08.480 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:35:08.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440108 2023-09-23T03:35:08.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440108 -le 1695440506 ]] 2023-09-23T03:35:08.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:35:08.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:35:08.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:18.498 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:35:18.504 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:35:18.504 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:35:18.505 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:35:19.674 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:35:19.676 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:35:19.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440119 2023-09-23T03:35:19.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440119 -le 1695440506 ]] 2023-09-23T03:35:19.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:35:19.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:35:19.677 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:29.688 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:35:29.689 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:35:29.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:35:29.734 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:35:30.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:35:30.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:35:30.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440130 2023-09-23T03:35:30.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440130 -le 1695440506 ]] 2023-09-23T03:35:30.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:35:30.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:35:30.861 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:40.869 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:35:42.507 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:35:42.507 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:35:42.507 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:35:42.507 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:35:42.507 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:35:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440141 2023-09-23T03:35:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440141 -le 1695440506 ]] 2023-09-23T03:35:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:35:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:35:42.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:51.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:35:51.924 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:35:51.934 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:35:51.934 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:35:53.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:35:53.229 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:35:53.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440153 2023-09-23T03:35:53.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440153 -le 1695440506 ]] 2023-09-23T03:35:53.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:35:53.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:35:53.231 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:35:54.320 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:35:54.320 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:35:54.320 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:35:54.321 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:35:54.319+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:36:03.236 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:03.239 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:03.239 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:03.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:36:04.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:36:04.468 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:36:04.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440164 2023-09-23T03:36:04.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440164 -le 1695440506 ]] 2023-09-23T03:36:04.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:36:04.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:36:04.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:36:14.480 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:14.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:14.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:36:14.492 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:15.505 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:36:15.505 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:36:15.506 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440175 2023-09-23T03:36:15.506 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440175 -le 1695440506 ]] 2023-09-23T03:36:15.506 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:36:15.506 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:36:15.506 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:36:25.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:25.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:25.511 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:25.511 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:36:26.961 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:36:26.968 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:36:26.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440186 2023-09-23T03:36:26.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440186 -le 1695440506 ]] 2023-09-23T03:36:26.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:36:26.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:36:26.992 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:36:37.000 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:36:37.003 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:37.013 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:37.032 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:38.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:36:38.301 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:36:38.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440198 2023-09-23T03:36:38.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440198 -le 1695440506 ]] 2023-09-23T03:36:38.301 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:36:38.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:36:38.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:36:48.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:48.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:48.307 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:36:48.308 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:49.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:36:49.577 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:36:49.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440209 2023-09-23T03:36:49.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440209 -le 1695440506 ]] 2023-09-23T03:36:49.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:36:49.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:36:49.579 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:36:59.587 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:36:59.587 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:36:59.589 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:36:59.606 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:00.892 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:00.896 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:00.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440220 2023-09-23T03:37:00.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440220 -le 1695440506 ]] 2023-09-23T03:37:00.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:00.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:00.897 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:37:10.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:10.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:37:10.914 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:37:10.916 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:37:12.123 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:12.123 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:12.124 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440232 2023-09-23T03:37:12.124 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440232 -le 1695440506 ]] 2023-09-23T03:37:12.124 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:12.124 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:12.124 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:37:22.126 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:22.126 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:37:22.130 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:37:22.134 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:37:23.346 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:23.346 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:23.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440243 2023-09-23T03:37:23.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440243 -le 1695440506 ]] 2023-09-23T03:37:23.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:23.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:23.348 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:37:33.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:37:33.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:37:33.351 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:33.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:37:34.678 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:34.679 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:34.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440254 2023-09-23T03:37:34.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440254 -le 1695440506 ]] 2023-09-23T03:37:34.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:34.679 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:34.680 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:37:44.693 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:37:44.699 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:44.701 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:37:44.705 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:37:46.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:46.036 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:46.043 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440266 2023-09-23T03:37:46.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440266 -le 1695440506 ]] 2023-09-23T03:37:46.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:46.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:46.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:37:56.050 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:37:56.051 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:37:56.058 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:37:56.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:37:57.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:37:57.459 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:37:57.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440277 2023-09-23T03:37:57.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440277 -le 1695440506 ]] 2023-09-23T03:37:57.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:37:57.467 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:37:57.468 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:38:07.469 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:38:07.469 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:38:07.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:38:07.477 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:38:08.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:38:08.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:38:08.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440288 2023-09-23T03:38:08.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440288 -le 1695440506 ]] 2023-09-23T03:38:08.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:38:08.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:38:08.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:38:18.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:38:18.810 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:38:18.812 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:38:18.813 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:38:20.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:38:20.342 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:38:20.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440300 2023-09-23T03:38:20.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440300 -le 1695440506 ]] 2023-09-23T03:38:20.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:38:20.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:38:20.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:38:30.358 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:38:30.362 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:38:30.367 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:38:30.371 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:38:31.599 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:38:31.600 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:38:31.605 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440311 2023-09-23T03:38:31.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440311 -le 1695440506 ]] 2023-09-23T03:38:31.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:38:31.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:38:31.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:38:41.613 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:38:41.615 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:38:41.616 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:38:41.623 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:38:42.806 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:38:42.806 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:38:42.807 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440322 2023-09-23T03:38:42.807 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440322 -le 1695440506 ]] 2023-09-23T03:38:42.807 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:38:42.807 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:38:42.807 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:38:52.810 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:38:52.811 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:38:52.811 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:38:52.827 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:38:54.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:38:54.030 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:38:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440334 2023-09-23T03:38:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440334 -le 1695440506 ]] 2023-09-23T03:38:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:38:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:38:54.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:39:02.068 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:02.070+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:39:02.073 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:02.070+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:39:02.073 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:02.070+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:39:02.073 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:02.070+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:39:02.073 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:02.070+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:39:04.038 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:39:04.038 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:39:04.041 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:39:04.047 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:39:04.992 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:04.990+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:39:05.007 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.006+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:39:05.008 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.010+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:39:05.016 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.018+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:39:05.017 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.018+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:39:05.017 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.018+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:39:05.018 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.018+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:39:05.018 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.018+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:39:05.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.022+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:39:05.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.022+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:39:05.024 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.022+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:39:05.034 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.034+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:39:05.035 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.034+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:39:05.035 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.034+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:39:05.035 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:39:05.034+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:39:05.439 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:39:05.439 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:39:05.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440345 2023-09-23T03:39:05.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440345 -le 1695440506 ]] 2023-09-23T03:39:05.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:39:05.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:39:05.440 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:39:15.442 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:39:15.442 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:39:15.442 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:39:15.461 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:39:16.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:39:16.700 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:39:16.702 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440356 2023-09-23T03:39:16.702 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440356 -le 1695440506 ]] 2023-09-23T03:39:16.703 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:39:16.703 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:39:16.703 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:39:26.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:39:26.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:39:26.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:39:26.707 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:39:27.828 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:39:27.829 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:39:27.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440367 2023-09-23T03:39:27.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440367 -le 1695440506 ]] 2023-09-23T03:39:27.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:39:27.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:39:27.830 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:39:37.832 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:39:37.832 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:39:37.833 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:39:37.837 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:39:39.183 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:39:39.184 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:39:39.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440379 2023-09-23T03:39:39.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440379 -le 1695440506 ]] 2023-09-23T03:39:39.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:39:39.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:39:39.185 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:39:49.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:39:49.196 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:39:49.196 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:39:49.225 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:39:50.610 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:39:50.610 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:39:50.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440390 2023-09-23T03:39:50.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440390 -le 1695440506 ]] 2023-09-23T03:39:50.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:39:50.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:39:50.612 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:00.623 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:00.629 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:00.635 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:00.637 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:01.848 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:01.851 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:01.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440401 2023-09-23T03:40:01.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440401 -le 1695440506 ]] 2023-09-23T03:40:01.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:01.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:01.853 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:11.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:11.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:11.868 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:11.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:13.303 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:13.304 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:13.305 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440413 2023-09-23T03:40:13.305 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440413 -le 1695440506 ]] 2023-09-23T03:40:13.305 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:13.305 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:13.305 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:23.313 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:23.318 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:23.318 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:23.318 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:24.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:24.759 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:24.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440424 2023-09-23T03:40:24.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440424 -le 1695440506 ]] 2023-09-23T03:40:24.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:24.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:24.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:28.427 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.424+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:40:28.431 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.432+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:40:28.432 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.432+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:40:28.439 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.436+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:40:28.439 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.436+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:40:28.441 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.440+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:40:28.441 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.440+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:40:28.441 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.440+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:40:28.442 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:28.440+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:40:32.320 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.316+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:40:32.327 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.320+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:40:32.329 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.324+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:40:32.329 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.324+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:40:32.352 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.344+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:40:32.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.344+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:40:32.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.344+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:40:32.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:40:32.344+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:40:34.776 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:34.776 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:34.776 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:34.776 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:35.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:35.834 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:35.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440435 2023-09-23T03:40:35.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440435 -le 1695440506 ]] 2023-09-23T03:40:35.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:35.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:35.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:45.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:45.851 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:45.851 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:45.852 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:46.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:46.909 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:46.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440446 2023-09-23T03:40:46.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440446 -le 1695440506 ]] 2023-09-23T03:40:46.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:46.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:46.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:40:56.914 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:40:56.922 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:40:56.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:40:56.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:40:58.234 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:40:58.234 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:40:58.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440458 2023-09-23T03:40:58.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440458 -le 1695440506 ]] 2023-09-23T03:40:58.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:40:58.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:40:58.235 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:41:08.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:41:08.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:41:08.259 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:41:08.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:41:09.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:41:09.321 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:41:09.322 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440469 2023-09-23T03:41:09.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440469 -le 1695440506 ]] 2023-09-23T03:41:09.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:41:09.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:41:09.323 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:41:19.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:41:19.328 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:41:19.328 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:41:19.331 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:41:20.526 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:41:20.527 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:41:20.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440480 2023-09-23T03:41:20.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440480 -le 1695440506 ]] 2023-09-23T03:41:20.527 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:41:20.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:41:20.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:41:30.533 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:41:30.533 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:41:30.535 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:41:30.548 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:41:31.846 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:41:31.855 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:41:31.856 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440491 2023-09-23T03:41:31.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440491 -le 1695440506 ]] 2023-09-23T03:41:31.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:41:31.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:41:31.857 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:41:40.676 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:41:40.675+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:41:41.859 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:41:41.860 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:41:41.863 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:41:41.886 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:41:43.064 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:41:43.076 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:41:43.077 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440503 2023-09-23T03:41:43.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440503 -le 1695440506 ]] 2023-09-23T03:41:43.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ ! -z 172.21.15.2:0/4250006422 ]] 2023-09-23T03:41:43.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 172.21.15.2:0/4250006422 != \1\7\2\.\2\1\.\1\5\.\2\:\0\/\4\2\5\0\0\0\6\4\2\2 ]] 2023-09-23T03:41:43.078 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:41:53.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ ceph mgr dump 2023-09-23T03:41:53.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq 'select(.name == "rbd_support")' 2023-09-23T03:41:53.092 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-09-23T03:41:53.098 INFO:tasks.workunit.client.0.smithi002.stderr:++ jq '.active_clients[]' 2023-09-23T03:41:54.341 INFO:tasks.workunit.client.0.smithi002.stderr:+ CLIENT_ADDR=172.21.15.2:0/4250006422 2023-09-23T03:41:54.341 INFO:tasks.workunit.client.0.smithi002.stderr:++ date +%s 2023-09-23T03:41:54.342 INFO:tasks.workunit.client.0.smithi002.stderr:+ CURRENT_TIME=1695440514 2023-09-23T03:41:54.342 INFO:tasks.workunit.client.0.smithi002.stderr:+ [[ 1695440514 -le 1695440506 ]] 2023-09-23T03:41:54.342 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i = 1 )) 2023-09-23T03:41:54.342 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:41:54.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:41:54.592 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:41:54.595+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:41:54.592 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:41:54.609 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:02.467 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.466+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:42:02.491 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:42:02.491 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:42:02.491 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:42:02.491 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:42:02.491 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:42:02.492 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:42:02.492 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:42:02.492 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:42:02.490+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:42:04.614 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:04.615 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:04.615 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:04.761 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:04.762+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:04.761 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:04.775 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:14.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:14.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:14.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:14.940 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:14.942+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:14.941 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:14.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:24.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:24.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:24.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:25.118 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:25.118+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:25.119 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:25.142 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:35.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:35.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:35.149 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:35.285 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:35.286+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:35.285 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:35.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:45.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:45.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:45.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:45.393 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:45.393+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:45.393 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:45.403 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:42:55.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:42:55.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:42:55.400 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:42:55.506 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:42:55.505+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:42:55.507 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:42:55.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:05.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:05.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:05.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:05.624 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:05.625+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:05.630 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:05.661 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:15.666 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:15.666 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:15.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:15.770 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:15.769+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:15.771 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:15.777 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:25.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:25.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:25.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:25.883 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:25.881+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:25.883 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:25.896 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:35.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:35.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:35.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:36.008 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:36.008+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:36.009 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:36.013 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:46.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:46.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:46.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:46.165 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:46.164+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:46.166 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:46.189 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:43:56.191 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:43:56.191 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:43:56.191 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:43:56.317 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:43:56.316+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:43:56.318 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:43:56.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:06.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:06.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:06.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:06.580 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:06.580+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:06.581 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:06.585 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:16.588 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:16.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:16.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:16.717 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:16.716+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:16.718 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:16.728 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:26.729 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:26.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:26.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:26.853 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:26.851+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:26.853 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:26.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:36.869 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:36.869 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:36.869 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:37.033 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:37.031+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:37.033 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:37.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:41.709 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:44:41.711+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:44:41.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:44:41.715+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:44:41.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:44:41.715+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:44:41.714 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:44:41.715+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:44:41.714 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:44:41.715+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:44:47.053 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:47.053 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:47.053 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:47.164 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:47.167 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:47.167+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:47.182 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:44:57.195 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:44:57.195 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:44:57.195 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:44:57.321 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:44:57.323+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:44:57.321 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:44:57.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:07.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:07.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:07.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:45:07.475 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:07.475+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:07.475 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:07.497 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:17.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:17.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:17.499 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:45:17.592 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:17.594+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:17.592 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:17.598 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:27.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:27.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:27.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:45:27.776 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:27.778+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:27.777 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:27.790 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:37.791 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:37.791 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:37.791 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:45:37.901 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:37.902+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:37.902 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:37.906 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:47.908 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:47.908 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:47.908 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-09-23T03:45:48.097 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:48.098+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:48.097 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:48.110 INFO:tasks.workunit.client.0.smithi002.stderr:+ sleep 10 2023-09-23T03:45:58.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i++ )) 2023-09-23T03:45:58.113 INFO:tasks.workunit.client.0.smithi002.stderr:+ (( i <= 24 )) 2023-09-23T03:45:58.113 INFO:tasks.workunit.client.0.smithi002.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-09-23T03:45:58.113 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'every 2m' 2023-09-23T03:45:58.255 INFO:tasks.ceph.mgr.x.smithi002.stderr:2023-09-23T03:45:58.258+0000 7f399021e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-09-23T03:45:58.256 INFO:tasks.workunit.client.0.smithi002.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-09-23T03:45:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T03:45:58.279 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-09-23T03:45:58.279 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-23T03:45:59.192 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:45:59.185+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:45:59.209 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:45:59.209+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:45:59.209 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:45:59.209+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:45:59.209 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:45:59.209+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:46:00.054 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-23T03:46:00.123 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e7d75845e58c41f0816122297fc17e70 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-09-23T03:46:00.127 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-23T03:46:00.170 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-23T03:46:00.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-23T03:46:00.179 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:00.930 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:00.931 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:00.978 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2137,"stamp":"2023-09-23T03:46:00.158419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20202129940,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1092296,"num_read_kb":3803278,"num_write":962744,"num_write_kb":3748649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83676,"ondisk_log_size":83676,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38442360,"kb_used_data":34104480,"kb_used_omap":3749,"kb_used_meta":4334106,"kb_avail":244673160,"statfs":{"total":289910292480,"available":250545315840,"internally_reserved":0,"allocated":34922987520,"data_stored":98803392387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4438125390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,16,8,20,10,19,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8990720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2274,"num_read_kb":9020,"num_write":2210,"num_write_kb":8840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005470"},"pg_stats":[{"pgid":"2.7","version":"150'7563","reported_seq":12314,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.840841+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T03:45:54.840841+0000","last_peered":"2023-09-23T03:45:54.840841+0000","last_clean":"2023-09-23T03:45:54.840841+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:45:54.840841+0000","last_undegraded":"2023-09-23T03:45:54.840841+0000","last_fullsized":"2023-09-23T03:45:54.840841+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:24:11.771899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'5206","reported_seq":8999,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:52.840569+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T03:45:52.840569+0000","last_peered":"2023-09-23T03:45:52.840569+0000","last_clean":"2023-09-23T03:45:52.840569+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:45:52.840569+0000","last_undegraded":"2023-09-23T03:45:52.840569+0000","last_fullsized":"2023-09-23T03:45:52.840569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:12:08.842822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'6542","reported_seq":11245,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.850194+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T03:45:54.850194+0000","last_peered":"2023-09-23T03:45:54.850194+0000","last_clean":"2023-09-23T03:45:54.850194+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:45:54.850194+0000","last_undegraded":"2023-09-23T03:45:54.850194+0000","last_fullsized":"2023-09-23T03:45:54.850194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:27:35.102921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'9088","reported_seq":16547,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.856093+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T03:45:54.856093+0000","last_peered":"2023-09-23T03:45:54.856093+0000","last_clean":"2023-09-23T03:45:54.856093+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:45:54.856093+0000","last_undegraded":"2023-09-23T03:45:54.856093+0000","last_fullsized":"2023-09-23T03:45:54.856093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:59:18.637875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'234327","reported_seq":465602,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:55.768388+0000","last_change":"2023-09-23T03:17:11.825145+0000","last_active":"2023-09-23T03:45:55.768388+0000","last_peered":"2023-09-23T03:45:55.768388+0000","last_clean":"2023-09-23T03:45:55.768388+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:45:55.768388+0000","last_undegraded":"2023-09-23T03:45:55.768388+0000","last_fullsized":"2023-09-23T03:45:55.768388+0000","mapping_epoch":18,"log_start":"147'224278","ondisk_log_start":"147'224278","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:46:28.840179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5035393024,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":234026,"num_read_kb":915444,"num_write":231303,"num_write_kb":925212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'1902","reported_seq":2212,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:49.965278+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T03:44:49.965278+0000","last_peered":"2023-09-23T03:44:49.965278+0000","last_clean":"2023-09-23T03:44:49.965278+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:44:49.965278+0000","last_undegraded":"2023-09-23T03:44:49.965278+0000","last_fullsized":"2023-09-23T03:44:49.965278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1902,"log_dups_size":0,"ondisk_log_size":1902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'245108","reported_seq":486969,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:56.442543+0000","last_change":"2023-09-23T03:17:11.830613+0000","last_active":"2023-09-23T03:45:56.442543+0000","last_peered":"2023-09-23T03:45:56.442543+0000","last_clean":"2023-09-23T03:45:56.442543+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:45:56.442543+0000","last_undegraded":"2023-09-23T03:45:56.442543+0000","last_fullsized":"2023-09-23T03:45:56.442543+0000","mapping_epoch":18,"log_start":"148'235061","ondisk_log_start":"148'235061","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:28:29.986344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5265526784,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":244775,"num_read_kb":957164,"num_write":241927,"num_write_kb":967708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"146'2112","reported_seq":5557,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963442+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T03:45:57.963442+0000","last_peered":"2023-09-23T03:45:57.963442+0000","last_clean":"2023-09-23T03:45:57.963442+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:45:57.963442+0000","last_undegraded":"2023-09-23T03:45:57.963442+0000","last_fullsized":"2023-09-23T03:45:57.963442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:24:33.920017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'239125","reported_seq":475081,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:55.770476+0000","last_change":"2023-09-23T03:17:11.825445+0000","last_active":"2023-09-23T03:45:55.770476+0000","last_peered":"2023-09-23T03:45:55.770476+0000","last_clean":"2023-09-23T03:45:55.770476+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:45:55.770476+0000","last_undegraded":"2023-09-23T03:45:55.770476+0000","last_fullsized":"2023-09-23T03:45:55.770476+0000","mapping_epoch":18,"log_start":"147'229122","ondisk_log_start":"147'229122","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:55:47.773993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5132406784,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":238766,"num_read_kb":934692,"num_write":236041,"num_write_kb":944164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"146'1479","reported_seq":4737,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963726+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T03:45:57.963726+0000","last_peered":"2023-09-23T03:45:57.963726+0000","last_clean":"2023-09-23T03:45:57.963726+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:45:57.963726+0000","last_undegraded":"2023-09-23T03:45:57.963726+0000","last_fullsized":"2023-09-23T03:45:57.963726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:01:51.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'222078","reported_seq":441055,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:56.465200+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:45:56.465200+0000","last_peered":"2023-09-23T03:45:56.465200+0000","last_clean":"2023-09-23T03:45:56.465200+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:45:56.465200+0000","last_undegraded":"2023-09-23T03:45:56.465200+0000","last_fullsized":"2023-09-23T03:45:56.465200+0000","mapping_epoch":18,"log_start":"147'212036","ondisk_log_start":"147'212036","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:23:23.627622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4768198675,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":221614,"num_read_kb":867628,"num_write":219167,"num_write_kb":876662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'9546","reported_seq":17314,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.945286+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T03:45:54.945286+0000","last_peered":"2023-09-23T03:45:54.945286+0000","last_clean":"2023-09-23T03:45:54.945286+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:45:54.945286+0000","last_undegraded":"2023-09-23T03:45:54.945286+0000","last_fullsized":"2023-09-23T03:45:54.945286+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:40:20.715457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":306,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:20.444574+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T03:44:20.444574+0000","last_peered":"2023-09-23T03:44:20.444574+0000","last_clean":"2023-09-23T03:44:20.444574+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:44:20.444574+0000","last_undegraded":"2023-09-23T03:44:20.444574+0000","last_fullsized":"2023-09-23T03:44:20.444574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20201525267,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":939181,"num_read_kb":3674928,"num_write":928438,"num_write_kb":3713746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34916769792,"data_stored":98800173056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43438,"ondisk_log_size":43438,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13385368,"kb_used_data":11368848,"kb_used_omap":1243,"kb_used_meta":2015268,"kb_avail":80986472,"statfs":{"total":96636764160,"available":82930147328,"internally_reserved":0,"allocated":11641700352,"data_stored":32934063610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":2063634604},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.63400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12502628,"kb_used_data":11368540,"kb_used_omap":1255,"kb_used_meta":1132824,"kb_avail":81869212,"statfs":{"total":96636764160,"available":83834073088,"internally_reserved":0,"allocated":11641384960,"data_stored":32934679340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1160012625},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,3,5,5,13,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.95999999999999996},"min":{"1min":0.20300000000000001,"5min":0.161,"15min":0.161},"max":{"1min":3.923,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.79900000000000004},"min":{"1min":0.24299999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.5570000000000004,"5min":4.9299999999999997,"15min":10.307},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Sep 23 03:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.57899999999999996,"15min":0.625},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":3.952,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.251},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.74399999999999999,"15min":0.626},"min":{"1min":0.247,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.1909999999999998,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.30099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739161,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12554364,"kb_used_data":11367092,"kb_used_omap":1250,"kb_used_meta":1186013,"kb_avail":81817476,"statfs":{"total":96636764160,"available":83781095424,"internally_reserved":0,"allocated":11639902208,"data_stored":32934649437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1214478161},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,5,13,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11637985280,"data_stored":32933367808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11638497280,"data_stored":32933404672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11640287232,"data_stored":32933400576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:00.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:01.908 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:01.964 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2137,"stamp":"2023-09-23T03:46:00.158419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20202129940,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1092296,"num_read_kb":3803278,"num_write":962744,"num_write_kb":3748649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83676,"ondisk_log_size":83676,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38442360,"kb_used_data":34104480,"kb_used_omap":3749,"kb_used_meta":4334106,"kb_avail":244673160,"statfs":{"total":289910292480,"available":250545315840,"internally_reserved":0,"allocated":34922987520,"data_stored":98803392387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4438125390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,16,8,20,10,19,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8990720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2274,"num_read_kb":9020,"num_write":2210,"num_write_kb":8840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005470"},"pg_stats":[{"pgid":"2.7","version":"150'7563","reported_seq":12314,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.840841+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T03:45:54.840841+0000","last_peered":"2023-09-23T03:45:54.840841+0000","last_clean":"2023-09-23T03:45:54.840841+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:45:54.840841+0000","last_undegraded":"2023-09-23T03:45:54.840841+0000","last_fullsized":"2023-09-23T03:45:54.840841+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:24:11.771899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'5206","reported_seq":8999,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:52.840569+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T03:45:52.840569+0000","last_peered":"2023-09-23T03:45:52.840569+0000","last_clean":"2023-09-23T03:45:52.840569+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:45:52.840569+0000","last_undegraded":"2023-09-23T03:45:52.840569+0000","last_fullsized":"2023-09-23T03:45:52.840569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:12:08.842822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'6542","reported_seq":11245,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.850194+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T03:45:54.850194+0000","last_peered":"2023-09-23T03:45:54.850194+0000","last_clean":"2023-09-23T03:45:54.850194+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:45:54.850194+0000","last_undegraded":"2023-09-23T03:45:54.850194+0000","last_fullsized":"2023-09-23T03:45:54.850194+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:27:35.102921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'9088","reported_seq":16547,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.856093+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T03:45:54.856093+0000","last_peered":"2023-09-23T03:45:54.856093+0000","last_clean":"2023-09-23T03:45:54.856093+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:45:54.856093+0000","last_undegraded":"2023-09-23T03:45:54.856093+0000","last_fullsized":"2023-09-23T03:45:54.856093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:59:18.637875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'234327","reported_seq":465602,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:55.768388+0000","last_change":"2023-09-23T03:17:11.825145+0000","last_active":"2023-09-23T03:45:55.768388+0000","last_peered":"2023-09-23T03:45:55.768388+0000","last_clean":"2023-09-23T03:45:55.768388+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:45:55.768388+0000","last_undegraded":"2023-09-23T03:45:55.768388+0000","last_fullsized":"2023-09-23T03:45:55.768388+0000","mapping_epoch":18,"log_start":"147'224278","ondisk_log_start":"147'224278","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:46:28.840179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5035393024,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":234026,"num_read_kb":915444,"num_write":231303,"num_write_kb":925212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'1902","reported_seq":2212,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:49.965278+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T03:44:49.965278+0000","last_peered":"2023-09-23T03:44:49.965278+0000","last_clean":"2023-09-23T03:44:49.965278+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:44:49.965278+0000","last_undegraded":"2023-09-23T03:44:49.965278+0000","last_fullsized":"2023-09-23T03:44:49.965278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1902,"log_dups_size":0,"ondisk_log_size":1902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'245108","reported_seq":486969,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:56.442543+0000","last_change":"2023-09-23T03:17:11.830613+0000","last_active":"2023-09-23T03:45:56.442543+0000","last_peered":"2023-09-23T03:45:56.442543+0000","last_clean":"2023-09-23T03:45:56.442543+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:45:56.442543+0000","last_undegraded":"2023-09-23T03:45:56.442543+0000","last_fullsized":"2023-09-23T03:45:56.442543+0000","mapping_epoch":18,"log_start":"148'235061","ondisk_log_start":"148'235061","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:28:29.986344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5265526784,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":244775,"num_read_kb":957164,"num_write":241927,"num_write_kb":967708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"146'2112","reported_seq":5557,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963442+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T03:45:57.963442+0000","last_peered":"2023-09-23T03:45:57.963442+0000","last_clean":"2023-09-23T03:45:57.963442+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:45:57.963442+0000","last_undegraded":"2023-09-23T03:45:57.963442+0000","last_fullsized":"2023-09-23T03:45:57.963442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:24:33.920017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'239125","reported_seq":475081,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:55.770476+0000","last_change":"2023-09-23T03:17:11.825445+0000","last_active":"2023-09-23T03:45:55.770476+0000","last_peered":"2023-09-23T03:45:55.770476+0000","last_clean":"2023-09-23T03:45:55.770476+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:45:55.770476+0000","last_undegraded":"2023-09-23T03:45:55.770476+0000","last_fullsized":"2023-09-23T03:45:55.770476+0000","mapping_epoch":18,"log_start":"147'229122","ondisk_log_start":"147'229122","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:55:47.773993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5132406784,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":238766,"num_read_kb":934692,"num_write":236041,"num_write_kb":944164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"146'1479","reported_seq":4737,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963726+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T03:45:57.963726+0000","last_peered":"2023-09-23T03:45:57.963726+0000","last_clean":"2023-09-23T03:45:57.963726+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:45:57.963726+0000","last_undegraded":"2023-09-23T03:45:57.963726+0000","last_fullsized":"2023-09-23T03:45:57.963726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:01:51.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'222078","reported_seq":441055,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:56.465200+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:45:56.465200+0000","last_peered":"2023-09-23T03:45:56.465200+0000","last_clean":"2023-09-23T03:45:56.465200+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:45:56.465200+0000","last_undegraded":"2023-09-23T03:45:56.465200+0000","last_fullsized":"2023-09-23T03:45:56.465200+0000","mapping_epoch":18,"log_start":"147'212036","ondisk_log_start":"147'212036","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:23:23.627622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4768198675,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":221614,"num_read_kb":867628,"num_write":219167,"num_write_kb":876662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'9546","reported_seq":17314,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:54.945286+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T03:45:54.945286+0000","last_peered":"2023-09-23T03:45:54.945286+0000","last_clean":"2023-09-23T03:45:54.945286+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:45:54.945286+0000","last_undegraded":"2023-09-23T03:45:54.945286+0000","last_fullsized":"2023-09-23T03:45:54.945286+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:40:20.715457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":306,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:20.444574+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T03:44:20.444574+0000","last_peered":"2023-09-23T03:44:20.444574+0000","last_clean":"2023-09-23T03:44:20.444574+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:44:20.444574+0000","last_undegraded":"2023-09-23T03:44:20.444574+0000","last_fullsized":"2023-09-23T03:44:20.444574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20201525267,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":939181,"num_read_kb":3674928,"num_write":928438,"num_write_kb":3713746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34916769792,"data_stored":98800173056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40141,"ondisk_log_size":40141,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43438,"ondisk_log_size":43438,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13385368,"kb_used_data":11368848,"kb_used_omap":1243,"kb_used_meta":2015268,"kb_avail":80986472,"statfs":{"total":96636764160,"available":82930147328,"internally_reserved":0,"allocated":11641700352,"data_stored":32934063610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":2063634604},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.63400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12502628,"kb_used_data":11368540,"kb_used_omap":1255,"kb_used_meta":1132824,"kb_avail":81869212,"statfs":{"total":96636764160,"available":83834073088,"internally_reserved":0,"allocated":11641384960,"data_stored":32934679340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1160012625},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,8,3,5,5,13,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.95999999999999996},"min":{"1min":0.20300000000000001,"5min":0.161,"15min":0.161},"max":{"1min":3.923,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.79900000000000004},"min":{"1min":0.24299999999999999,"5min":0.185,"15min":0.185},"max":{"1min":4.5570000000000004,"5min":4.9299999999999997,"15min":10.307},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Sep 23 03:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.57899999999999996,"15min":0.625},"min":{"1min":0.29499999999999998,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":3.952,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.251},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.74399999999999999,"15min":0.626},"min":{"1min":0.247,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.1909999999999998,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.30099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739161,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12554364,"kb_used_data":11367092,"kb_used_omap":1250,"kb_used_meta":1186013,"kb_avail":81817476,"statfs":{"total":96636764160,"available":83781095424,"internally_reserved":0,"allocated":11639902208,"data_stored":32934649437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1214478161},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6,5,13,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11637985280,"data_stored":32933367808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11638497280,"data_stored":32933404672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11640287232,"data_stored":32933400576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:01.965 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-23T03:46:01.966 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:02.809 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:02.864 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2138,"stamp":"2023-09-23T03:46:02.158768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20205959700,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1093288,"num_read_kb":3807226,"num_write":963686,"num_write_kb":3752417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83833,"ondisk_log_size":83833,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38475280,"kb_used_data":34111032,"kb_used_omap":3749,"kb_used_meta":4360474,"kb_avail":244640240,"statfs":{"total":289910292480,"available":250511605760,"internally_reserved":0,"allocated":34929696768,"data_stored":98803408771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4465126222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,14,14,18,26,22,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12820480,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3266,"num_read_kb":12968,"num_write":3152,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005465"},"pg_stats":[{"pgid":"2.7","version":"150'7567","reported_seq":12321,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:59.193501+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T03:45:59.193501+0000","last_peered":"2023-09-23T03:45:59.193501+0000","last_clean":"2023-09-23T03:45:59.193501+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:45:59.193501+0000","last_undegraded":"2023-09-23T03:45:59.193501+0000","last_fullsized":"2023-09-23T03:45:59.193501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:24:11.771899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'5208","reported_seq":9002,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:59.163970+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T03:45:59.163970+0000","last_peered":"2023-09-23T03:45:59.163970+0000","last_clean":"2023-09-23T03:45:59.163970+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:45:59.163970+0000","last_undegraded":"2023-09-23T03:45:59.163970+0000","last_fullsized":"2023-09-23T03:45:59.163970+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:12:08.842822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'6546","reported_seq":11252,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:59.176768+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T03:45:59.176768+0000","last_peered":"2023-09-23T03:45:59.176768+0000","last_clean":"2023-09-23T03:45:59.176768+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:45:59.176768+0000","last_undegraded":"2023-09-23T03:45:59.176768+0000","last_fullsized":"2023-09-23T03:45:59.176768+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:27:35.102921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'9093","reported_seq":16556,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:59.221889+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T03:45:59.221889+0000","last_peered":"2023-09-23T03:45:59.221889+0000","last_clean":"2023-09-23T03:45:59.221889+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:45:59.221889+0000","last_undegraded":"2023-09-23T03:45:59.221889+0000","last_fullsized":"2023-09-23T03:45:59.221889+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:59:18.637875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'234564","reported_seq":466088,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:00.877369+0000","last_change":"2023-09-23T03:17:11.825145+0000","last_active":"2023-09-23T03:46:00.877369+0000","last_peered":"2023-09-23T03:46:00.877369+0000","last_clean":"2023-09-23T03:46:00.877369+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:46:00.877369+0000","last_undegraded":"2023-09-23T03:46:00.877369+0000","last_fullsized":"2023-09-23T03:46:00.877369+0000","mapping_epoch":18,"log_start":"147'224478","ondisk_log_start":"147'224478","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:46:28.840179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5036351488,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":234275,"num_read_kb":916436,"num_write":231540,"num_write_kb":926160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'1902","reported_seq":2212,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:49.965278+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T03:44:49.965278+0000","last_peered":"2023-09-23T03:44:49.965278+0000","last_clean":"2023-09-23T03:44:49.965278+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:44:49.965278+0000","last_undegraded":"2023-09-23T03:44:49.965278+0000","last_fullsized":"2023-09-23T03:44:49.965278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1902,"log_dups_size":0,"ondisk_log_size":1902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'245334","reported_seq":487436,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:01.615181+0000","last_change":"2023-09-23T03:17:11.830613+0000","last_active":"2023-09-23T03:46:01.615181+0000","last_peered":"2023-09-23T03:46:01.615181+0000","last_clean":"2023-09-23T03:46:01.615181+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:46:01.615181+0000","last_undegraded":"2023-09-23T03:46:01.615181+0000","last_fullsized":"2023-09-23T03:46:01.615181+0000","mapping_epoch":18,"log_start":"148'235262","ondisk_log_start":"148'235262","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:28:29.986344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5266448384,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":245012,"num_read_kb":958108,"num_write":242153,"num_write_kb":968612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"146'2112","reported_seq":5557,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963442+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T03:45:57.963442+0000","last_peered":"2023-09-23T03:45:57.963442+0000","last_clean":"2023-09-23T03:45:57.963442+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:45:57.963442+0000","last_undegraded":"2023-09-23T03:45:57.963442+0000","last_fullsized":"2023-09-23T03:45:57.963442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:24:33.920017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'239361","reported_seq":475551,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:00.794484+0000","last_change":"2023-09-23T03:17:11.825445+0000","last_active":"2023-09-23T03:46:00.794484+0000","last_peered":"2023-09-23T03:46:00.794484+0000","last_clean":"2023-09-23T03:46:00.794484+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:46:00.794484+0000","last_undegraded":"2023-09-23T03:46:00.794484+0000","last_fullsized":"2023-09-23T03:46:00.794484+0000","mapping_epoch":18,"log_start":"148'229324","ondisk_log_start":"148'229324","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:55:47.773993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5133365248,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":239001,"num_read_kb":935620,"num_write":236277,"num_write_kb":945108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"146'1479","reported_seq":4737,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:57.963726+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T03:45:57.963726+0000","last_peered":"2023-09-23T03:45:57.963726+0000","last_clean":"2023-09-23T03:45:57.963726+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:45:57.963726+0000","last_undegraded":"2023-09-23T03:45:57.963726+0000","last_fullsized":"2023-09-23T03:45:57.963726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:01:51.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'222321","reported_seq":441566,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:01.604263+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:46:01.604263+0000","last_peered":"2023-09-23T03:46:01.604263+0000","last_clean":"2023-09-23T03:46:01.604263+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:46:01.604263+0000","last_undegraded":"2023-09-23T03:46:01.604263+0000","last_fullsized":"2023-09-23T03:46:01.604263+0000","mapping_epoch":18,"log_start":"147'212239","ondisk_log_start":"147'212239","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:23:23.627622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4769189907,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":221885,"num_read_kb":868712,"num_write":219410,"num_write_kb":877634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'9552","reported_seq":17325,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:45:59.268937+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T03:45:59.268937+0000","last_peered":"2023-09-23T03:45:59.268937+0000","last_clean":"2023-09-23T03:45:59.268937+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:45:59.268937+0000","last_undegraded":"2023-09-23T03:45:59.268937+0000","last_fullsized":"2023-09-23T03:45:59.268937+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:40:20.715457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":306,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:20.444574+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T03:44:20.444574+0000","last_peered":"2023-09-23T03:44:20.444574+0000","last_clean":"2023-09-23T03:44:20.444574+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:44:20.444574+0000","last_undegraded":"2023-09-23T03:44:20.444574+0000","last_fullsized":"2023-09-23T03:44:20.444574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20205355027,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":940173,"num_read_kb":3678876,"num_write":929380,"num_write_kb":3717514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34923474944,"data_stored":98800209920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40277,"ondisk_log_size":40277,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43459,"ondisk_log_size":43459,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13385368,"kb_used_data":11368848,"kb_used_omap":1243,"kb_used_meta":2015268,"kb_avail":80986472,"statfs":{"total":96636764160,"available":82930147328,"internally_reserved":0,"allocated":11641700352,"data_stored":32934063610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":2063634604},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.63400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12509900,"kb_used_data":11371652,"kb_used_omap":1255,"kb_used_meta":1136984,"kb_avail":81861940,"statfs":{"total":96636764160,"available":83826626560,"internally_reserved":0,"allocated":11644571648,"data_stored":32934671148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1164272465},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,13,8,9,17,17,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.877,"15min":0.98099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":3.7869999999999999,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.008,"5min":0.75600000000000001,"15min":0.82599999999999996},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":10.307},"last":0.29099999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":4.8129999999999997,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.79800000000000004,"15min":0.67500000000000004},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17499999999999999},"max":{"1min":4.8330000000000002,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739162,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12580012,"kb_used_data":11370532,"kb_used_omap":1250,"kb_used_meta":1208221,"kb_avail":81791828,"statfs":{"total":96636764160,"available":83754831872,"internally_reserved":0,"allocated":11643424768,"data_stored":32934674013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1237219153},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,9,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.311},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11641360384,"data_stored":32933400576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11641827328,"data_stored":32933408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11640287232,"data_stored":32933400576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:02.866 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-23T03:46:03.721 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:03.722 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":150,"fsid":"f964ea34-ba1c-42d5-92d6-b55c00c696eb","created":"2023-09-23T02:39:32.280358+0000","modified":"2023-09-23T03:45:18.676411+0000","last_up_change":"2023-09-23T02:39:41.587263+0000","last_in_change":"2023-09-23T02:39:33.183919+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-23T02:39:43.121371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-23T02:39:47.647096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-09-23T02:39:55.309261+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":108,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"274b1544-bb02-4626-b8d0-a991f832cfa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6801","nonce":474982742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6803","nonce":474982742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6807","nonce":474982742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":474982742},{"type":"v1","addr":"172.21.15.2:6805","nonce":474982742}]},"public_addr":"172.21.15.2:6801/474982742","cluster_addr":"172.21.15.2:6803/474982742","heartbeat_back_addr":"172.21.15.2:6807/474982742","heartbeat_front_addr":"172.21.15.2:6805/474982742","state":["exists","up"]},{"osd":1,"uuid":"dff4c364-8a65-4fdb-b791-b4707748ec65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6809","nonce":2376239628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6811","nonce":2376239628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6815","nonce":2376239628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2376239628},{"type":"v1","addr":"172.21.15.2:6813","nonce":2376239628}]},"public_addr":"172.21.15.2:6809/2376239628","cluster_addr":"172.21.15.2:6811/2376239628","heartbeat_back_addr":"172.21.15.2:6815/2376239628","heartbeat_front_addr":"172.21.15.2:6813/2376239628","state":["exists","up"]},{"osd":2,"uuid":"1f73dc35-18f2-49cc-af51-be0c48ebdc9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6817","nonce":363956679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6819","nonce":363956679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6823","nonce":363956679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":363956679},{"type":"v1","addr":"172.21.15.2:6821","nonce":363956679}]},"public_addr":"172.21.15.2:6817/363956679","cluster_addr":"172.21.15.2:6819/363956679","heartbeat_back_addr":"172.21.15.2:6823/363956679","heartbeat_front_addr":"172.21.15.2:6821/363956679","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.767365+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.221094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-23T02:39:38.758039+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/4250006422":"2023-09-23T04:26:28.096507+0000","172.21.15.2:0/1654913027":"2023-09-23T04:24:05.646794+0000","172.21.15.2:0/4188811661":"2023-09-23T04:19:32.145686+0000","172.21.15.2:0/4113803064":"2023-09-23T04:18:20.817901+0000","172.21.15.2:0/2146552285":"2023-09-23T04:21:43.455127+0000","172.21.15.2:0/2352963285":"2023-09-23T04:15:59.629127+0000","172.21.15.2:0/3079579092":"2023-09-23T03:53:25.943140+0000","172.21.15.2:0/2866803494":"2023-09-23T04:06:30.145925+0000","172.21.15.2:0/778467004":"2023-09-23T03:51:13.392655+0000","172.21.15.2:0/400934215":"2023-09-23T03:55:49.123452+0000","172.21.15.2:0/2884784619":"2023-09-23T03:48:51.652697+0000","172.21.15.2:0/2409967782":"2023-09-23T03:52:24.499568+0000","172.21.15.2:0/2718665785":"2023-09-23T03:46:28.483110+0000","172.21.15.2:0/1878591735":"2023-09-23T04:08:53.542578+0000","172.21.15.2:0/2891944060":"2023-09-23T04:20:43.327067+0000","172.21.15.2:0/3523592144":"2023-09-23T03:58:11.748396+0000","172.21.15.2:0/2953468242":"2023-09-23T03:50:02.620482+0000","172.21.15.2:0/1377367264":"2023-09-23T04:05:19.323571+0000","172.21.15.2:0/3586892185":"2023-09-23T04:13:38.773288+0000","172.21.15.2:0/1376584732":"2023-09-23T03:57:01.179672+0000","172.21.15.2:0/2003455134":"2023-09-23T03:54:37.555999+0000","172.21.15.2:0/1286535523":"2023-09-23T04:04:07.741852+0000","172.21.15.2:0/1409823025":"2023-09-23T03:47:39.915815+0000","172.21.15.2:0/2045148581":"2023-09-23T04:00:34.658595+0000","172.21.15.2:0/312273758":"2023-09-23T03:59:22.785153+0000","172.21.15.2:0/860360033":"2023-09-23T04:11:15.890621+0000","172.21.15.2:0/166270078":"2023-09-23T04:22:54.419543+0000","172.21.15.2:0/4137551135":"2023-09-23T04:01:45.047542+0000","172.21.15.2:0/1944658510":"2023-09-23T04:17:10.147412+0000","172.21.15.2:0/4140630340":"2023-09-23T04:02:56.497964+0000","172.21.15.2:0/3873857838":"2023-09-23T04:07:41.846041+0000","172.21.15.2:0/1770443901":"2023-09-23T04:09:53.296622+0000","172.21.15.2:0/2531452558":"2023-09-23T04:25:16.273686+0000","172.21.15.2:0/1510231876":"2023-09-23T04:12:27.288201+0000","172.21.15.2:0/3739782250":"2023-09-23T04:14:49.351460+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-23T03:46:04.808 INFO:tasks.ceph:Scrubbing osd.0 2023-09-23T03:46:04.808 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-23T03:46:05.210 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-09-23T03:46:05.210 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T03:46:05.210 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-09-23T03:46:05.261 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-23T03:46:06.094 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 0 to deep-scrub 2023-09-23T03:46:06.139 INFO:tasks.ceph:Scrubbing osd.1 2023-09-23T03:46:06.139 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-23T03:46:06.575 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-09-23T03:46:06.575 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T03:46:06.576 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-09-23T03:46:06.650 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-23T03:46:07.433 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 1 to deep-scrub 2023-09-23T03:46:07.481 INFO:tasks.ceph:Scrubbing osd.2 2023-09-23T03:46:07.481 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-23T03:46:07.827 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-09-23T03:46:07.827 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-23T03:46:07.828 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-09-23T03:46:07.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-23T03:46:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 2 to deep-scrub 2023-09-23T03:46:08.862 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:09.930 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:09.931 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:10.016 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2141,"stamp":"2023-09-23T03:46:08.159989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20209953300,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1094300,"num_read_kb":3811242,"num_write":964668,"num_write_kb":3756345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83715,"ondisk_log_size":83715,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37775888,"kb_used_data":34120820,"kb_used_omap":3749,"kb_used_meta":3651290,"kb_avail":245339632,"statfs":{"total":289910292480,"available":251227783168,"internally_reserved":0,"allocated":34939719680,"data_stored":98803531651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":3738921806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,11,5,8,13,16,15],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10223616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2572,"num_read_kb":10224,"num_write":2513,"num_write_kb":10052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005703"},"pg_stats":[{"pgid":"2.7","version":"150'7570","reported_seq":12327,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.079299+0000","last_change":"2023-09-23T02:39:49.246411+0000","last_active":"2023-09-23T03:46:05.079299+0000","last_peered":"2023-09-23T03:46:05.079299+0000","last_clean":"2023-09-23T03:46:05.079299+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:46:05.079299+0000","last_undegraded":"2023-09-23T03:46:05.079299+0000","last_fullsized":"2023-09-23T03:46:05.079299+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:24:11.771899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'5209","reported_seq":9004,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:03.080374+0000","last_change":"2023-09-23T02:39:49.245775+0000","last_active":"2023-09-23T03:46:03.080374+0000","last_peered":"2023-09-23T03:46:03.080374+0000","last_clean":"2023-09-23T03:46:03.080374+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:46:03.080374+0000","last_undegraded":"2023-09-23T03:46:03.080374+0000","last_fullsized":"2023-09-23T03:46:03.080374+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:12:08.842822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'6549","reported_seq":11258,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.065570+0000","last_change":"2023-09-23T02:39:49.245589+0000","last_active":"2023-09-23T03:46:05.065570+0000","last_peered":"2023-09-23T03:46:05.065570+0000","last_clean":"2023-09-23T03:46:05.065570+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:46:05.065570+0000","last_undegraded":"2023-09-23T03:46:05.065570+0000","last_fullsized":"2023-09-23T03:46:05.065570+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:27:35.102921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'9097","reported_seq":16564,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.098383+0000","last_change":"2023-09-23T02:39:49.247422+0000","last_active":"2023-09-23T03:46:05.098383+0000","last_peered":"2023-09-23T03:46:05.098383+0000","last_clean":"2023-09-23T03:46:05.098383+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:46:05.098383+0000","last_undegraded":"2023-09-23T03:46:05.098383+0000","last_fullsized":"2023-09-23T03:46:05.098383+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:59:18.637875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'234828","reported_seq":466584,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.915155+0000","last_change":"2023-09-23T03:17:11.825145+0000","last_active":"2023-09-23T03:46:05.915155+0000","last_peered":"2023-09-23T03:46:05.915155+0000","last_clean":"2023-09-23T03:46:05.915155+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:46:05.915155+0000","last_undegraded":"2023-09-23T03:46:05.915155+0000","last_fullsized":"2023-09-23T03:46:05.915155+0000","mapping_epoch":18,"log_start":"148'224786","ondisk_log_start":"148'224786","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T02:46:28.840179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5037420544,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":234511,"num_read_kb":917372,"num_write":231804,"num_write_kb":927216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"132'1902","reported_seq":2212,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:49.965278+0000","last_change":"2023-09-23T02:39:49.243830+0000","last_active":"2023-09-23T03:44:49.965278+0000","last_peered":"2023-09-23T03:44:49.965278+0000","last_clean":"2023-09-23T03:44:49.965278+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:44:49.965278+0000","last_undegraded":"2023-09-23T03:44:49.965278+0000","last_fullsized":"2023-09-23T03:44:49.965278+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1902,"log_dups_size":0,"ondisk_log_size":1902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T11:31:55.062820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'245580","reported_seq":487958,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:06.691721+0000","last_change":"2023-09-23T03:17:11.830613+0000","last_active":"2023-09-23T03:46:06.691721+0000","last_peered":"2023-09-23T03:46:06.691721+0000","last_clean":"2023-09-23T03:46:06.691721+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:46:06.691721+0000","last_undegraded":"2023-09-23T03:46:06.691721+0000","last_fullsized":"2023-09-23T03:46:06.691721+0000","mapping_epoch":18,"log_start":"148'235566","ondisk_log_start":"148'235566","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T14:28:29.986344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5267464192,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":245286,"num_read_kb":959200,"num_write":242399,"num_write_kb":969596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"146'2112","reported_seq":5559,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:03.096334+0000","last_change":"2023-09-23T02:39:49.249197+0000","last_active":"2023-09-23T03:46:03.096334+0000","last_peered":"2023-09-23T03:46:03.096334+0000","last_clean":"2023-09-23T03:46:03.096334+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:46:03.096334+0000","last_undegraded":"2023-09-23T03:46:03.096334+0000","last_fullsized":"2023-09-23T03:46:03.096334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:24:33.920017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'239603","reported_seq":476042,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.888348+0000","last_change":"2023-09-23T03:17:11.825445+0000","last_active":"2023-09-23T03:46:05.888348+0000","last_peered":"2023-09-23T03:46:05.888348+0000","last_clean":"2023-09-23T03:46:05.888348+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:46:05.888348+0000","last_undegraded":"2023-09-23T03:46:05.888348+0000","last_fullsized":"2023-09-23T03:46:05.888348+0000","mapping_epoch":18,"log_start":"148'229527","ondisk_log_start":"148'229527","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T13:55:47.773993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5134348288,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":239251,"num_read_kb":936604,"num_write":236519,"num_write_kb":946076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"146'1479","reported_seq":4739,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:03.080220+0000","last_change":"2023-09-23T02:39:49.248939+0000","last_active":"2023-09-23T03:46:03.080220+0000","last_peered":"2023-09-23T03:46:03.080220+0000","last_clean":"2023-09-23T03:46:03.080220+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:46:03.080220+0000","last_undegraded":"2023-09-23T03:46:03.080220+0000","last_fullsized":"2023-09-23T03:46:03.080220+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:01:51.571064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'222551","reported_seq":442052,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:06.700234+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:46:06.700234+0000","last_peered":"2023-09-23T03:46:06.700234+0000","last_clean":"2023-09-23T03:46:06.700234+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:46:06.700234+0000","last_undegraded":"2023-09-23T03:46:06.700234+0000","last_fullsized":"2023-09-23T03:46:06.700234+0000","mapping_epoch":18,"log_start":"147'212540","ondisk_log_start":"147'212540","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T10:23:23.627622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4770115603,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":222137,"num_read_kb":869716,"num_write":219640,"num_write_kb":878554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'9557","reported_seq":17335,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:05.134004+0000","last_change":"2023-09-23T02:39:49.246511+0000","last_active":"2023-09-23T03:46:05.134004+0000","last_peered":"2023-09-23T03:46:05.134004+0000","last_clean":"2023-09-23T03:46:05.134004+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:46:05.134004+0000","last_undegraded":"2023-09-23T03:46:05.134004+0000","last_fullsized":"2023-09-23T03:46:05.134004+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:48.217692+0000","last_clean_scrub_stamp":"2023-09-23T02:39:48.217692+0000","objects_scrubbed":0,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:40:20.715457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":306,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-09-23T03:44:20.444574+0000","last_change":"2023-09-23T02:39:44.795409+0000","last_active":"2023-09-23T03:44:20.444574+0000","last_peered":"2023-09-23T03:44:20.444574+0000","last_clean":"2023-09-23T03:44:20.444574+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:44:20.444574+0000","last_undegraded":"2023-09-23T03:44:20.444574+0000","last_fullsized":"2023-09-23T03:44:20.444574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:43.591005+0000","last_clean_scrub_stamp":"2023-09-23T02:39:43.591005+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:23:56.063753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20209348627,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":941185,"num_read_kb":3682892,"num_write":930362,"num_write_kb":3721442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34933735424,"data_stored":98800291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40143,"ondisk_log_size":40143,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43475,"ondisk_log_size":43475,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739167,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12651588,"kb_used_data":11371964,"kb_used_omap":1243,"kb_used_meta":1278372,"kb_avail":81720252,"statfs":{"total":96636764160,"available":83681538048,"internally_reserved":0,"allocated":11644891136,"data_stored":32934079994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1309053100},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":2.403},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":0.24099999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.29699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739166,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12535456,"kb_used_data":11374996,"kb_used_omap":1255,"kb_used_meta":1159192,"kb_avail":81836384,"statfs":{"total":96636764160,"available":83800457216,"internally_reserved":0,"allocated":11647995904,"data_stored":32934744876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1187013457},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,4,7,11,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.877,"15min":0.98099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":3.7869999999999999,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.008,"5min":0.75600000000000001,"15min":0.82599999999999996},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":10.307},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":4.8129999999999997,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.37},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.79800000000000004,"15min":0.67500000000000004},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17499999999999999},"max":{"1min":4.8330000000000002,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.25900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739163,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12588844,"kb_used_data":11373860,"kb_used_omap":1250,"kb_used_meta":1213725,"kb_avail":81782996,"statfs":{"total":96636764160,"available":83745787904,"internally_reserved":0,"allocated":11646832640,"data_stored":32934706781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1242855249},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,4,4,6,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":0.247}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.5},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11644805120,"data_stored":32933457920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11645329408,"data_stored":32933408768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11643600896,"data_stored":32933425152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:10.019 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.019 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.020 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.021 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.021 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.021 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.021 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-23T02:39:48.217692+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=48, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.021 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-23T02:39:43.591005+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=43, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:10.022 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:46:25.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:46:25.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:46:25.568 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:46:25.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:46:25.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:46:25.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:46:25.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:46:25.569 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:46:25.569+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:46:30.023 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:30.724 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2153,"stamp":"2023-09-23T03:46:30.169395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20227021332,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1098504,"num_read_kb":3827902,"num_write":968804,"num_write_kb":3772889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84119,"ondisk_log_size":84119,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":37977304,"kb_used_data":34166652,"kb_used_omap":3749,"kb_used_meta":3806874,"kb_avail":245138216,"statfs":{"total":289910292480,"available":251021533184,"internally_reserved":0,"allocated":34986651648,"data_stored":98804162435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":3898239822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,4,6,12,17,21,16,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8654848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2079,"num_read_kb":8244,"num_write":2100,"num_write_kb":8400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":14747,"num_omap_keys":254,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001739"},"pg_stats":[{"pgid":"2.7","version":"150'7617","reported_seq":12396,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:25.558423+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:46:25.558423+0000","last_peered":"2023-09-23T03:46:25.558423+0000","last_clean":"2023-09-23T03:46:25.558423+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:46:25.558423+0000","last_undegraded":"2023-09-23T03:46:25.558423+0000","last_fullsized":"2023-09-23T03:46:25.558423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7617,"log_dups_size":0,"ondisk_log_size":7617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'5245","reported_seq":9054,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:24.526599+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:46:24.526599+0000","last_peered":"2023-09-23T03:46:24.526599+0000","last_clean":"2023-09-23T03:46:24.526599+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:46:24.526599+0000","last_undegraded":"2023-09-23T03:46:24.526599+0000","last_fullsized":"2023-09-23T03:46:24.526599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'6592","reported_seq":11323,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:25.565551+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:46:25.565551+0000","last_peered":"2023-09-23T03:46:25.565551+0000","last_clean":"2023-09-23T03:46:25.565551+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:46:25.565551+0000","last_undegraded":"2023-09-23T03:46:25.565551+0000","last_fullsized":"2023-09-23T03:46:25.565551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6592,"log_dups_size":0,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'9144","reported_seq":16637,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:25.596332+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:46:25.596332+0000","last_peered":"2023-09-23T03:46:25.596332+0000","last_clean":"2023-09-23T03:46:25.596332+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:46:25.596332+0000","last_undegraded":"2023-09-23T03:46:25.596332+0000","last_fullsized":"2023-09-23T03:46:25.596332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9144,"log_dups_size":0,"ondisk_log_size":9144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"150'235855","reported_seq":468683,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:46:26.124205+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:46:26.124205+0000","last_peered":"2023-09-23T03:46:26.124205+0000","last_clean":"2023-09-23T03:46:26.124205+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:46:26.124205+0000","last_undegraded":"2023-09-23T03:46:26.124205+0000","last_fullsized":"2023-09-23T03:46:26.124205+0000","mapping_epoch":18,"log_start":"148'225794","ondisk_log_start":"148'225794","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":142,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5041590272,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":235565,"num_read_kb":921540,"num_write":232831,"num_write_kb":931324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2248,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:09.370089+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:09.370089+0000","last_peered":"2023-09-23T03:46:09.370089+0000","last_clean":"2023-09-23T03:46:09.370089+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:09.370089+0000","last_undegraded":"2023-09-23T03:46:09.370089+0000","last_fullsized":"2023-09-23T03:46:09.370089+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"150'246608","reported_seq":490106,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:26.973274+0000","last_change":"2023-09-23T03:17:11.830613+0000","last_active":"2023-09-23T03:46:26.973274+0000","last_peered":"2023-09-23T03:46:26.973274+0000","last_clean":"2023-09-23T03:46:26.973274+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:46:26.973274+0000","last_undegraded":"2023-09-23T03:46:26.973274+0000","last_fullsized":"2023-09-23T03:46:26.973274+0000","mapping_epoch":18,"log_start":"148'236576","ondisk_log_start":"148'236576","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5271777280,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":246403,"num_read_kb":963632,"num_write":243427,"num_write_kb":973708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5602,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.688493+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:46:28.688493+0000","last_peered":"2023-09-23T03:46:28.688493+0000","last_clean":"2023-09-23T03:46:28.688493+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:46:28.688493+0000","last_undegraded":"2023-09-23T03:46:28.688493+0000","last_fullsized":"2023-09-23T03:46:28.688493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'240685","reported_seq":478176,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:46:26.103419+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:46:26.103419+0000","last_peered":"2023-09-23T03:46:26.103419+0000","last_clean":"2023-09-23T03:46:26.103419+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:46:26.103419+0000","last_undegraded":"2023-09-23T03:46:26.103419+0000","last_fullsized":"2023-09-23T03:46:26.103419+0000","mapping_epoch":18,"log_start":"148'230638","ondisk_log_start":"148'230638","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":96,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 10s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5138800640,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":240294,"num_read_kb":940752,"num_write":237601,"num_write_kb":950404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4787,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.738146+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:46:28.738146+0000","last_peered":"2023-09-23T03:46:28.738146+0000","last_clean":"2023-09-23T03:46:28.738146+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:46:28.738146+0000","last_undegraded":"2023-09-23T03:46:28.738146+0000","last_fullsized":"2023-09-23T03:46:28.738146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"150'223550","reported_seq":444038,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:26.973970+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:46:26.973970+0000","last_peered":"2023-09-23T03:46:26.973970+0000","last_clean":"2023-09-23T03:46:26.973970+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:46:26.973970+0000","last_undegraded":"2023-09-23T03:46:26.973970+0000","last_fullsized":"2023-09-23T03:46:26.973970+0000","mapping_epoch":18,"log_start":"148'213450","ondisk_log_start":"148'213450","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4774248467,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":223127,"num_read_kb":873628,"num_write":220639,"num_write_kb":882550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"150'9614","reported_seq":17422,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:26.095349+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:46:26.095349+0000","last_peered":"2023-09-23T03:46:26.095349+0000","last_clean":"2023-09-23T03:46:26.095349+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:46:26.095349+0000","last_undegraded":"2023-09-23T03:46:26.095349+0000","last_fullsized":"2023-09-23T03:46:26.095349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9614,"log_dups_size":0,"ondisk_log_size":9614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":318,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:08.627854+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:08.627854+0000","last_peered":"2023-09-23T03:46:08.627854+0000","last_clean":"2023-09-23T03:46:08.627854+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:08.627854+0000","last_undegraded":"2023-09-23T03:46:08.627854+0000","last_fullsized":"2023-09-23T03:46:08.627854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20226416659,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":945389,"num_read_kb":3699552,"num_write":934498,"num_write_kb":3737986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":34980380672,"data_stored":98800951296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40240,"ondisk_log_size":40240,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43780,"ondisk_log_size":43780,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739172,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12665800,"kb_used_data":11389312,"kb_used_omap":1243,"kb_used_meta":1275236,"kb_avail":81706040,"statfs":{"total":96636764160,"available":83666984960,"internally_reserved":0,"allocated":11662655488,"data_stored":32934333946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1305841836},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":2.3519999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":2.3639999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":2.1589999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739170,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12645472,"kb_used_data":11388888,"kb_used_omap":1255,"kb_used_meta":1255320,"kb_avail":81726368,"statfs":{"total":96636764160,"available":83687800832,"internally_reserved":0,"allocated":11662221312,"data_stored":32934912812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1285448529},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,3,9,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.877,"15min":0.98099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":3.7869999999999999,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.008,"5min":0.75600000000000001,"15min":0.82599999999999996},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":10.307},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":4.8129999999999997,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.499},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.79800000000000004,"15min":0.67500000000000004},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17499999999999999},"max":{"1min":4.8330000000000002,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739167,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12666032,"kb_used_data":11388452,"kb_used_omap":1250,"kb_used_meta":1276317,"kb_avail":81705808,"statfs":{"total":96636764160,"available":83666747392,"internally_reserved":0,"allocated":11661774848,"data_stored":32934915677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1306949457},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,7,8,18,15,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":4.8929999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11659161600,"data_stored":32933605376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11659788288,"data_stored":32933662720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11661430784,"data_stored":32933683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:30.782 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:30.783 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:30.783 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:30.783 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:30.783 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:46:50.784 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:46:51.774 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:46:51.775 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:46:51.823 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2163,"stamp":"2023-09-23T03:46:50.175304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20243532308,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1102531,"num_read_kb":3843866,"num_write":972797,"num_write_kb":3788861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84092,"ondisk_log_size":84092,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38220976,"kb_used_data":34207760,"kb_used_omap":3749,"kb_used_meta":4009434,"kb_avail":244894544,"statfs":{"total":289910292480,"available":250772013056,"internally_reserved":0,"allocated":35028746240,"data_stored":98804719094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4105661262},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,14,11,16,13,8,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8155136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1994,"num_read_kb":7892,"num_write":1966,"num_write_kb":7864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004487"},"pg_stats":[{"pgid":"2.7","version":"151'7623","reported_seq":12415,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:44.039009+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:46:44.039009+0000","last_peered":"2023-09-23T03:46:44.039009+0000","last_clean":"2023-09-23T03:46:44.039009+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:46:44.039009+0000","last_undegraded":"2023-09-23T03:46:44.039009+0000","last_fullsized":"2023-09-23T03:46:44.039009+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'5247","reported_seq":9061,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:38.916313+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:46:38.916313+0000","last_peered":"2023-09-23T03:46:38.916313+0000","last_clean":"2023-09-23T03:46:38.916313+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:46:38.916313+0000","last_undegraded":"2023-09-23T03:46:38.916313+0000","last_fullsized":"2023-09-23T03:46:38.916313+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'6598","reported_seq":11342,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:44.038803+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:46:44.038803+0000","last_peered":"2023-09-23T03:46:44.038803+0000","last_clean":"2023-09-23T03:46:44.038803+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:46:44.038803+0000","last_undegraded":"2023-09-23T03:46:44.038803+0000","last_fullsized":"2023-09-23T03:46:44.038803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6598,"log_dups_size":0,"ondisk_log_size":6598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'9152","reported_seq":16662,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:44.039237+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:46:44.039237+0000","last_peered":"2023-09-23T03:46:44.039237+0000","last_clean":"2023-09-23T03:46:44.039237+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:46:44.039237+0000","last_undegraded":"2023-09-23T03:46:44.039237+0000","last_fullsized":"2023-09-23T03:46:44.039237+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9152,"log_dups_size":0,"ondisk_log_size":9152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'236859","reported_seq":470715,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:46:46.448525+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:46:46.448525+0000","last_peered":"2023-09-23T03:46:46.448525+0000","last_clean":"2023-09-23T03:46:46.448525+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:46:46.448525+0000","last_undegraded":"2023-09-23T03:46:46.448525+0000","last_fullsized":"2023-09-23T03:46:46.448525+0000","mapping_epoch":18,"log_start":"148'226814","ondisk_log_start":"148'226814","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":397,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 36s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5045723136,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":236575,"num_read_kb":925548,"num_write":233835,"num_write_kb":935340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2249,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:09.370163+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:09.370163+0000","last_peered":"2023-09-23T03:46:09.370163+0000","last_clean":"2023-09-23T03:46:09.370163+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:09.370163+0000","last_undegraded":"2023-09-23T03:46:09.370163+0000","last_fullsized":"2023-09-23T03:46:09.370163+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'247632","reported_seq":492164,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:46:47.171357+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:46:47.171357+0000","last_peered":"2023-09-23T03:46:47.171357+0000","last_clean":"2023-09-23T03:46:47.171357+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:46:47.171357+0000","last_undegraded":"2023-09-23T03:46:47.171357+0000","last_fullsized":"2023-09-23T03:46:47.171357+0000","mapping_epoch":18,"log_start":"148'237584","ondisk_log_start":"148'237584","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":166,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 19s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5276061696,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":247421,"num_read_kb":967668,"num_write":244451,"num_write_kb":977804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5611,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:49.171554+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:46:49.171554+0000","last_peered":"2023-09-23T03:46:49.171554+0000","last_clean":"2023-09-23T03:46:49.171554+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:46:49.171554+0000","last_undegraded":"2023-09-23T03:46:49.171554+0000","last_fullsized":"2023-09-23T03:46:49.171554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'241685","reported_seq":480264,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:46:46.441985+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:46:46.441985+0000","last_peered":"2023-09-23T03:46:46.441985+0000","last_clean":"2023-09-23T03:46:46.441985+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:46:46.441985+0000","last_undegraded":"2023-09-23T03:46:46.441985+0000","last_fullsized":"2023-09-23T03:46:46.441985+0000","mapping_epoch":18,"log_start":"148'231650","ondisk_log_start":"148'231650","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":365,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 30s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5142917120,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":241364,"num_read_kb":944996,"num_write":238601,"num_write_kb":954404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4795,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:49.168008+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:46:49.168008+0000","last_peered":"2023-09-23T03:46:49.168008+0000","last_clean":"2023-09-23T03:46:49.168008+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:46:49.168008+0000","last_undegraded":"2023-09-23T03:46:49.168008+0000","last_fullsized":"2023-09-23T03:46:49.168008+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'224515","reported_seq":445937,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:47.156669+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:46:47.156669+0000","last_peered":"2023-09-23T03:46:47.156669+0000","last_clean":"2023-09-23T03:46:47.156669+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:46:47.156669+0000","last_undegraded":"2023-09-23T03:46:47.156669+0000","last_fullsized":"2023-09-23T03:46:47.156669+0000","mapping_epoch":18,"log_start":"148'214462","ondisk_log_start":"148'214462","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4778225683,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":224056,"num_read_kb":877304,"num_write":221604,"num_write_kb":886410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'9624","reported_seq":17453,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:44.040027+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:46:44.040027+0000","last_peered":"2023-09-23T03:46:44.040027+0000","last_clean":"2023-09-23T03:46:44.040027+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:46:44.040027+0000","last_undegraded":"2023-09-23T03:46:44.040027+0000","last_fullsized":"2023-09-23T03:46:44.040027+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9624,"log_dups_size":0,"ondisk_log_size":9624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":319,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:21.919759+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:21.919759+0000","last_peered":"2023-09-23T03:46:21.919759+0000","last_clean":"2023-09-23T03:46:21.919759+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:21.919759+0000","last_undegraded":"2023-09-23T03:46:21.919759+0000","last_fullsized":"2023-09-23T03:46:21.919759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20242927635,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":949416,"num_read_kb":3715516,"num_write":938491,"num_write_kb":3753958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35022483456,"data_stored":98801610752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40181,"ondisk_log_size":40181,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43812,"ondisk_log_size":43812,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739176,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12792276,"kb_used_data":11403404,"kb_used_omap":1243,"kb_used_meta":1387620,"kb_avail":81579564,"statfs":{"total":96636764160,"available":83537473536,"internally_reserved":0,"allocated":11677085696,"data_stored":32934503115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1420923052},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.0289999999999999,"15min":1.022},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17399999999999999},"max":{"1min":5.7869999999999999,"5min":6.9189999999999996,"15min":22.789000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.59599999999999997,"15min":0.58199999999999996},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.19},"max":{"1min":4.9020000000000001,"5min":5.1349999999999998,"15min":5.1779999999999999},"last":0.372}]},{"osd":1,"last update":"Sat Sep 23 03:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.78400000000000003,"15min":0.59799999999999998},"min":{"1min":0.27100000000000002,"5min":0.214,"15min":0.191},"max":{"1min":3.867,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.253},{"interface":"front","average":{"1min":1.431,"5min":0.996,"15min":0.84999999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.7990000000000004,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.35299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739174,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12695092,"kb_used_data":11402408,"kb_used_omap":1255,"kb_used_meta":1291416,"kb_avail":81676748,"statfs":{"total":96636764160,"available":83636989952,"internally_reserved":0,"allocated":11676065792,"data_stored":32935131133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1322410833},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,3,8,5,2,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.877,"15min":0.98099999999999998},"min":{"1min":0.20599999999999999,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":3.7869999999999999,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.008,"5min":0.75600000000000001,"15min":0.82599999999999996},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":10.307},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:45:59 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":4.8129999999999997,"5min":5.5469999999999997,"15min":9.7080000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.79800000000000004,"15min":0.67500000000000004},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17499999999999999},"max":{"1min":4.8330000000000002,"5min":7.9210000000000003,"15min":10.313000000000001},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739171,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12733608,"kb_used_data":11401948,"kb_used_omap":1250,"kb_used_meta":1330397,"kb_avail":81638232,"statfs":{"total":96636764160,"available":83597549568,"internally_reserved":0,"allocated":11675594752,"data_stored":32935084846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1362327377},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,6,6,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.77200000000000002,"15min":0.86199999999999999},"min":{"1min":0.28100000000000003,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":1.4199999999999999,"5min":11.715,"15min":26.273},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.22500000000000001,"5min":0.192,"15min":0.188},"max":{"1min":2.278,"5min":12.058,"15min":12.058},"last":4.2830000000000004}]},{"osd":2,"last update":"Sat Sep 23 03:45:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53400000000000003,"15min":0.59599999999999997},"min":{"1min":0.25,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.88400000000000001,"5min":4.9720000000000004,"15min":25.635999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.56699999999999995,"15min":0.63200000000000001},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":0.78800000000000003,"5min":5.0119999999999996,"15min":25.666},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11673247744,"data_stored":32933830656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11673784320,"data_stored":32933900288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11675451392,"data_stored":32933879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:46:51.825 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:51.825 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:51.825 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:51.826 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:46:51.826 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:47:11.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:47:12.705 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:47:12.706 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:47:12.759 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2174,"stamp":"2023-09-23T03:47:12.179293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20262529556,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1107162,"num_read_kb":3862258,"num_write":977427,"num_write_kb":3807381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84258,"ondisk_log_size":84258,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38564180,"kb_used_data":34254008,"kb_used_omap":3749,"kb_used_meta":4306394,"kb_avail":244551340,"statfs":{"total":289910292480,"available":250420572160,"internally_reserved":0,"allocated":35076104192,"data_stored":98805181942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4409748302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,19,33,17,11,13,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10629120,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2548,"num_read_kb":10124,"num_write":2587,"num_write_kb":10348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002135"},"pg_stats":[{"pgid":"2.7","version":"151'7644","reported_seq":12448,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:06.945935+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:47:06.945935+0000","last_peered":"2023-09-23T03:47:06.945935+0000","last_clean":"2023-09-23T03:47:06.945935+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:47:06.945935+0000","last_undegraded":"2023-09-23T03:47:06.945935+0000","last_fullsized":"2023-09-23T03:47:06.945935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'5254","reported_seq":9072,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:04.516555+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:47:04.516555+0000","last_peered":"2023-09-23T03:47:04.516555+0000","last_clean":"2023-09-23T03:47:04.516555+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:47:04.516555+0000","last_undegraded":"2023-09-23T03:47:04.516555+0000","last_fullsized":"2023-09-23T03:47:04.516555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'6619","reported_seq":11375,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:06.960673+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:47:06.960673+0000","last_peered":"2023-09-23T03:47:06.960673+0000","last_clean":"2023-09-23T03:47:06.960673+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:47:06.960673+0000","last_undegraded":"2023-09-23T03:47:06.960673+0000","last_fullsized":"2023-09-23T03:47:06.960673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'9180","reported_seq":16707,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:06.976414+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:47:06.976414+0000","last_peered":"2023-09-23T03:47:06.976414+0000","last_clean":"2023-09-23T03:47:06.976414+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:47:06.976414+0000","last_undegraded":"2023-09-23T03:47:06.976414+0000","last_fullsized":"2023-09-23T03:47:06.976414+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9180,"log_dups_size":0,"ondisk_log_size":9180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'238114","reported_seq":473204,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:11.835453+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:47:11.835453+0000","last_peered":"2023-09-23T03:47:11.835453+0000","last_clean":"2023-09-23T03:47:11.835453+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:47:11.835453+0000","last_undegraded":"2023-09-23T03:47:11.835453+0000","last_fullsized":"2023-09-23T03:47:11.835453+0000","mapping_epoch":18,"log_start":"148'228027","ondisk_log_start":"148'228027","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":763,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 61s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5050871808,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":237788,"num_read_kb":930376,"num_write":235090,"num_write_kb":940360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2250,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.761066+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:28.761066+0000","last_peered":"2023-09-23T03:46:28.761066+0000","last_clean":"2023-09-23T03:46:28.761066+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:28.761066+0000","last_undegraded":"2023-09-23T03:46:28.761066+0000","last_fullsized":"2023-09-23T03:46:28.761066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'248707","reported_seq":494363,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:07.336510+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:47:07.336510+0000","last_peered":"2023-09-23T03:47:07.336510+0000","last_clean":"2023-09-23T03:47:07.336510+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:47:07.336510+0000","last_undegraded":"2023-09-23T03:47:07.336510+0000","last_fullsized":"2023-09-23T03:47:07.336510+0000","mapping_epoch":18,"log_start":"148'238610","ondisk_log_start":"148'238610","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":291,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5280436224,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":248531,"num_read_kb":972068,"num_write":245526,"num_write_kb":982104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5619,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:09.638461+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:47:09.638461+0000","last_peered":"2023-09-23T03:47:09.638461+0000","last_clean":"2023-09-23T03:47:09.638461+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:47:09.638461+0000","last_undegraded":"2023-09-23T03:47:09.638461+0000","last_fullsized":"2023-09-23T03:47:09.638461+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'243004","reported_seq":482882,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:11.818949+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:47:11.818949+0000","last_peered":"2023-09-23T03:47:11.818949+0000","last_clean":"2023-09-23T03:47:11.818949+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:47:11.818949+0000","last_undegraded":"2023-09-23T03:47:11.818949+0000","last_fullsized":"2023-09-23T03:47:11.818949+0000","mapping_epoch":18,"log_start":"148'232973","ondisk_log_start":"148'232973","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":570,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5148340224,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":242647,"num_read_kb":950108,"num_write":239920,"num_write_kb":959680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4801,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:04.531830+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:47:04.531830+0000","last_peered":"2023-09-23T03:47:04.531830+0000","last_clean":"2023-09-23T03:47:04.531830+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:47:04.531830+0000","last_undegraded":"2023-09-23T03:47:04.531830+0000","last_fullsized":"2023-09-23T03:47:04.531830+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'225496","reported_seq":447943,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:07.390619+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:47:07.390619+0000","last_peered":"2023-09-23T03:47:07.390619+0000","last_clean":"2023-09-23T03:47:07.390619+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:47:07.390619+0000","last_undegraded":"2023-09-23T03:47:07.390619+0000","last_fullsized":"2023-09-23T03:47:07.390619+0000","mapping_epoch":18,"log_start":"148'215476","ondisk_log_start":"148'215476","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4782276627,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":225081,"num_read_kb":881356,"num_write":222585,"num_write_kb":890334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'9659","reported_seq":17507,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:07.039771+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:47:07.039771+0000","last_peered":"2023-09-23T03:47:07.039771+0000","last_clean":"2023-09-23T03:47:07.039771+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:47:07.039771+0000","last_undegraded":"2023-09-23T03:47:07.039771+0000","last_fullsized":"2023-09-23T03:47:07.039771+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9659,"log_dups_size":0,"ondisk_log_size":9659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":319,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:21.919759+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:21.919759+0000","last_peered":"2023-09-23T03:46:21.919759+0000","last_clean":"2023-09-23T03:46:21.919759+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:21.919759+0000","last_undegraded":"2023-09-23T03:46:21.919759+0000","last_fullsized":"2023-09-23T03:46:21.919759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20261924883,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":954047,"num_read_kb":3733908,"num_write":943121,"num_write_kb":3772478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35069370368,"data_stored":98801967104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40235,"ondisk_log_size":40235,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43924,"ondisk_log_size":43924,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739180,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12878396,"kb_used_data":11417392,"kb_used_omap":1243,"kb_used_meta":1459748,"kb_avail":81493444,"statfs":{"total":96636764160,"available":83449286656,"internally_reserved":0,"allocated":11691409408,"data_stored":32934671051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1494782124},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1,"15min":1.0349999999999999},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.6560000000000001,"5min":6.6609999999999996,"15min":22.789000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53100000000000003,"15min":0.55600000000000005},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.3639999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.71799999999999997,"15min":0.58099999999999996},"min":{"1min":0.23300000000000001,"5min":0.214,"15min":0.191},"max":{"1min":1.8700000000000001,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.309},{"interface":"front","average":{"1min":0.504,"5min":0.94799999999999995,"15min":0.83599999999999997},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":2.1589999999999998,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.34599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739178,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12840188,"kb_used_data":11417076,"kb_used_omap":1255,"kb_used_meta":1421848,"kb_avail":81531652,"statfs":{"total":96636764160,"available":83488411648,"internally_reserved":0,"allocated":11691085824,"data_stored":32935221245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1455973201},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,10,12,2,5,12,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.97499999999999998,"15min":0.995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":5.9029999999999996,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":10.307},"last":0.46200000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":3.327,"5min":4.8129999999999997,"15min":9.7080000000000002},"last":0.435},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.69299999999999995,"15min":0.67500000000000004},"min":{"1min":0.25900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":0.52200000000000002,"5min":5.5960000000000001,"15min":10.313000000000001},"last":0.69499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739176,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12845596,"kb_used_data":11419540,"kb_used_omap":1250,"kb_used_meta":1424797,"kb_avail":81526244,"statfs":{"total":96636764160,"available":83482873856,"internally_reserved":0,"allocated":11693608960,"data_stored":32935289646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1458992977},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,21,15,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.55600000000000005,"15min":0.754},"min":{"1min":0.24099999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":0.68600000000000005,"5min":3.988,"15min":11.715},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.73299999999999998,"15min":0.95299999999999996},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":4.2830000000000004,"5min":5.9349999999999996,"15min":12.058},"last":1.2190000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44,"15min":0.48299999999999998},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":1.331,"5min":1.591,"15min":4.9720000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.46899999999999997,"15min":0.52900000000000003},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":8.6750000000000007},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11691393024,"data_stored":32934014976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11688177664,"data_stored":32933965824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11689799680,"data_stored":32933986304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:47:12.760 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:12.761 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:12.761 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:12.761 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:12.761 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:47:32.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:47:33.533 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:47:33.534 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:47:33.570 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2184,"stamp":"2023-09-23T03:47:32.182867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20279396884,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1111210,"num_read_kb":3878330,"num_write":981489,"num_write_kb":3823629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84285,"ondisk_log_size":84285,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38840680,"kb_used_data":34295632,"kb_used_omap":3749,"kb_used_meta":4541274,"kb_avail":244274840,"statfs":{"total":289910292480,"available":250137436160,"internally_reserved":0,"allocated":35118727168,"data_stored":98805865974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4650265422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,12,12,16,20,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10428416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2516,"num_read_kb":10008,"num_write":2521,"num_write_kb":10084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002160"},"pg_stats":[{"pgid":"2.7","version":"151'7650","reported_seq":12466,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:25.000840+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:47:25.000840+0000","last_peered":"2023-09-23T03:47:25.000840+0000","last_clean":"2023-09-23T03:47:25.000840+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:47:25.000840+0000","last_undegraded":"2023-09-23T03:47:25.000840+0000","last_fullsized":"2023-09-23T03:47:25.000840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'5256","reported_seq":9078,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:19.879501+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:47:19.879501+0000","last_peered":"2023-09-23T03:47:19.879501+0000","last_clean":"2023-09-23T03:47:19.879501+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:47:19.879501+0000","last_undegraded":"2023-09-23T03:47:19.879501+0000","last_fullsized":"2023-09-23T03:47:19.879501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'6625","reported_seq":11393,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:25.000593+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:47:25.000593+0000","last_peered":"2023-09-23T03:47:25.000593+0000","last_clean":"2023-09-23T03:47:25.000593+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:47:25.000593+0000","last_undegraded":"2023-09-23T03:47:25.000593+0000","last_fullsized":"2023-09-23T03:47:25.000593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'9188","reported_seq":16731,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:25.000633+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:47:25.000633+0000","last_peered":"2023-09-23T03:47:25.000633+0000","last_clean":"2023-09-23T03:47:25.000633+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:47:25.000633+0000","last_undegraded":"2023-09-23T03:47:25.000633+0000","last_fullsized":"2023-09-23T03:47:25.000633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9188,"log_dups_size":0,"ondisk_log_size":9188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"151'239064","reported_seq":475149,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:32.090245+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:47:32.090245+0000","last_peered":"2023-09-23T03:47:32.090245+0000","last_clean":"2023-09-23T03:47:32.090245+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:47:32.090245+0000","last_undegraded":"2023-09-23T03:47:32.090245+0000","last_fullsized":"2023-09-23T03:47:32.090245+0000","mapping_epoch":18,"log_start":"149'229045","ondisk_log_start":"149'229045","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1044,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 82s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5054754816,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":238767,"num_read_kb":934276,"num_write":236040,"num_write_kb":944160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2250,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.761066+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:28.761066+0000","last_peered":"2023-09-23T03:46:28.761066+0000","last_clean":"2023-09-23T03:46:28.761066+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:28.761066+0000","last_undegraded":"2023-09-23T03:46:28.761066+0000","last_fullsized":"2023-09-23T03:46:28.761066+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"151'249776","reported_seq":496565,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:27.605475+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:47:27.605475+0000","last_peered":"2023-09-23T03:47:27.605475+0000","last_clean":"2023-09-23T03:47:27.605475+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:47:27.605475+0000","last_undegraded":"2023-09-23T03:47:27.605475+0000","last_fullsized":"2023-09-23T03:47:27.605475+0000","mapping_epoch":18,"log_start":"149'239726","ondisk_log_start":"149'239726","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":469,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 60s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5284823040,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":249657,"num_read_kb":976536,"num_write":246595,"num_write_kb":986380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5625,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:24.996676+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:47:24.996676+0000","last_peered":"2023-09-23T03:47:24.996676+0000","last_clean":"2023-09-23T03:47:24.996676+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:47:24.996676+0000","last_undegraded":"2023-09-23T03:47:24.996676+0000","last_fullsized":"2023-09-23T03:47:24.996676+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"151'244089","reported_seq":485006,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:32.076676+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:47:32.076676+0000","last_peered":"2023-09-23T03:47:32.076676+0000","last_clean":"2023-09-23T03:47:32.076676+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:47:32.076676+0000","last_undegraded":"2023-09-23T03:47:32.076676+0000","last_fullsized":"2023-09-23T03:47:32.076676+0000","mapping_epoch":18,"log_start":"149'233998","ondisk_log_start":"149'233998","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":738,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5152825344,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":243671,"num_read_kb":954156,"num_write":241005,"num_write_kb":964020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4809,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:24.999918+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:47:24.999918+0000","last_peered":"2023-09-23T03:47:24.999918+0000","last_clean":"2023-09-23T03:47:24.999918+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:47:24.999918+0000","last_undegraded":"2023-09-23T03:47:24.999918+0000","last_fullsized":"2023-09-23T03:47:24.999918+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"151'226454","reported_seq":449818,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:27.639410+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:47:27.639410+0000","last_peered":"2023-09-23T03:47:27.639410+0000","last_clean":"2023-09-23T03:47:27.639410+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:47:27.639410+0000","last_undegraded":"2023-09-23T03:47:27.639410+0000","last_fullsized":"2023-09-23T03:47:27.639410+0000","mapping_epoch":18,"log_start":"148'216384","ondisk_log_start":"148'216384","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4786389011,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":226000,"num_read_kb":885012,"num_write":223543,"num_write_kb":894166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"151'9669","reported_seq":17537,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:47:25.038811+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:47:25.038811+0000","last_peered":"2023-09-23T03:47:25.038811+0000","last_clean":"2023-09-23T03:47:25.038811+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:47:25.038811+0000","last_undegraded":"2023-09-23T03:47:25.038811+0000","last_fullsized":"2023-09-23T03:47:25.038811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9669,"log_dups_size":0,"ondisk_log_size":9669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":319,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-09-23T03:46:21.919759+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:21.919759+0000","last_peered":"2023-09-23T03:46:21.919759+0000","last_clean":"2023-09-23T03:46:21.919759+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:21.919759+0000","last_undegraded":"2023-09-23T03:46:21.919759+0000","last_fullsized":"2023-09-23T03:46:21.919759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20278792211,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":958095,"num_read_kb":3749980,"num_write":947183,"num_write_kb":3788726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35112321024,"data_stored":98802708480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40230,"ondisk_log_size":40230,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":43956,"ondisk_log_size":43956,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12982560,"kb_used_data":11431320,"kb_used_omap":1243,"kb_used_meta":1549988,"kb_avail":81389280,"statfs":{"total":96636764160,"available":83342622720,"internally_reserved":0,"allocated":11705671680,"data_stored":32934925003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1587187884},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1,"15min":1.0349999999999999},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.6560000000000001,"5min":6.6609999999999996,"15min":22.789000000000001},"last":0.5},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53100000000000003,"15min":0.55600000000000005},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.3639999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.27300000000000002}]},{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.71799999999999997,"15min":0.58099999999999996},"min":{"1min":0.23300000000000001,"5min":0.214,"15min":0.191},"max":{"1min":1.8700000000000001,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.94799999999999995,"15min":0.83599999999999997},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":2.1589999999999998,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739182,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12908172,"kb_used_data":11430980,"kb_used_omap":1255,"kb_used_meta":1475928,"kb_avail":81463668,"statfs":{"total":96636764160,"available":83418796032,"internally_reserved":0,"allocated":11705323520,"data_stored":32935462909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1511351121},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,1,4,3,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.97499999999999998,"15min":0.995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":5.9029999999999996,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.312},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":10.307},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":3.327,"5min":4.8129999999999997,"15min":9.7080000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.69299999999999995,"15min":0.67500000000000004},"min":{"1min":0.25900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":0.52200000000000002,"5min":5.5960000000000001,"15min":10.313000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739180,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12949948,"kb_used_data":11433332,"kb_used_omap":1250,"kb_used_meta":1515357,"kb_avail":81421892,"statfs":{"total":96636764160,"available":83376017408,"internally_reserved":0,"allocated":11707731968,"data_stored":32935478062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1551726417},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,4,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.55600000000000005,"15min":0.754},"min":{"1min":0.24099999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":0.68600000000000005,"5min":3.988,"15min":11.715},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.73299999999999998,"15min":0.95299999999999996},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":4.2830000000000004,"5min":5.9349999999999996,"15min":12.058},"last":0.378}]},{"osd":2,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44,"15min":0.48299999999999998},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":1.331,"5min":1.591,"15min":4.9720000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.46899999999999997,"15min":0.52900000000000003},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":8.6750000000000007},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11705643008,"data_stored":32934268928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11702591488,"data_stored":32934199296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11704086528,"data_stored":32934240256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:47:33.572 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:33.572 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:33.572 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:33.573 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:33.573 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:47:53.574 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:47:54.328 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:47:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:47:54.365 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2196,"stamp":"2023-09-23T03:47:54.190029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20298107412,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1115832,"num_read_kb":3896754,"num_write":985975,"num_write_kb":3841573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84299,"ondisk_log_size":84299,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39070156,"kb_used_data":34340016,"kb_used_omap":3749,"kb_used_meta":4726362,"kb_avail":244045364,"statfs":{"total":289910292480,"available":249902452736,"internally_reserved":0,"allocated":35164176384,"data_stored":98806557672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":4839795534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,10,18,12,20,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8347648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2091,"num_read_kb":8340,"num_write":1993,"num_write_kb":7972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.401241"},"pg_stats":[{"pgid":"2.7","version":"152'7668","reported_seq":12501,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:50.601371+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:47:50.601371+0000","last_peered":"2023-09-23T03:47:50.601371+0000","last_clean":"2023-09-23T03:47:50.601371+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:47:50.601371+0000","last_undegraded":"2023-09-23T03:47:50.601371+0000","last_fullsized":"2023-09-23T03:47:50.601371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7668,"log_dups_size":0,"ondisk_log_size":7668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'5262","reported_seq":9090,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:47.744329+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:47:47.744329+0000","last_peered":"2023-09-23T03:47:47.744329+0000","last_clean":"2023-09-23T03:47:47.744329+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:47:47.744329+0000","last_undegraded":"2023-09-23T03:47:47.744329+0000","last_fullsized":"2023-09-23T03:47:47.744329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'6643","reported_seq":11427,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:50.600603+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:47:50.600603+0000","last_peered":"2023-09-23T03:47:50.600603+0000","last_clean":"2023-09-23T03:47:50.600603+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:47:50.600603+0000","last_undegraded":"2023-09-23T03:47:50.600603+0000","last_fullsized":"2023-09-23T03:47:50.600603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6643,"log_dups_size":0,"ondisk_log_size":6643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'9212","reported_seq":16776,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:50.600545+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:47:50.600545+0000","last_peered":"2023-09-23T03:47:50.600545+0000","last_clean":"2023-09-23T03:47:50.600545+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:47:50.600545+0000","last_undegraded":"2023-09-23T03:47:50.600545+0000","last_fullsized":"2023-09-23T03:47:50.600545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9212,"log_dups_size":0,"ondisk_log_size":9212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'240035","reported_seq":477172,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:52.277405+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:47:52.277405+0000","last_peered":"2023-09-23T03:47:52.277405+0000","last_clean":"2023-09-23T03:47:52.277405+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:47:52.277405+0000","last_undegraded":"2023-09-23T03:47:52.277405+0000","last_fullsized":"2023-09-23T03:47:52.277405+0000","mapping_epoch":18,"log_start":"149'229957","ondisk_log_start":"149'229957","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1272,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5058801664,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":239799,"num_read_kb":938388,"num_write":237011,"num_write_kb":948044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2251,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:56.602443+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:56.602443+0000","last_peered":"2023-09-23T03:46:56.602443+0000","last_clean":"2023-09-23T03:46:56.602443+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:56.602443+0000","last_undegraded":"2023-09-23T03:46:56.602443+0000","last_fullsized":"2023-09-23T03:46:56.602443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'251082","reported_seq":499234,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:53.039240+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:47:53.039240+0000","last_peered":"2023-09-23T03:47:53.039240+0000","last_clean":"2023-09-23T03:47:53.039240+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:47:53.039240+0000","last_undegraded":"2023-09-23T03:47:53.039240+0000","last_fullsized":"2023-09-23T03:47:53.039240+0000","mapping_epoch":18,"log_start":"149'241059","ondisk_log_start":"149'241059","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":711,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 85s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5290262528,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":251000,"num_read_kb":981896,"num_write":247901,"num_write_kb":991604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5634,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:45.484847+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:47:45.484847+0000","last_peered":"2023-09-23T03:47:45.484847+0000","last_clean":"2023-09-23T03:47:45.484847+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:47:45.484847+0000","last_undegraded":"2023-09-23T03:47:45.484847+0000","last_fullsized":"2023-09-23T03:47:45.484847+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'245122","reported_seq":487106,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:47:52.256848+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:47:52.256848+0000","last_peered":"2023-09-23T03:47:52.256848+0000","last_clean":"2023-09-23T03:47:52.256848+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:47:52.256848+0000","last_undegraded":"2023-09-23T03:47:52.256848+0000","last_fullsized":"2023-09-23T03:47:52.256848+0000","mapping_epoch":18,"log_start":"149'235111","ondisk_log_start":"149'235111","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":931,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5157130240,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":244726,"num_read_kb":958360,"num_write":242038,"num_write_kb":968152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4818,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:45.476146+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:47:45.476146+0000","last_peered":"2023-09-23T03:47:45.476146+0000","last_clean":"2023-09-23T03:47:45.476146+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:47:45.476146+0000","last_undegraded":"2023-09-23T03:47:45.476146+0000","last_fullsized":"2023-09-23T03:47:45.476146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'227630","reported_seq":452189,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:53.048585+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:47:53.048585+0000","last_peered":"2023-09-23T03:47:53.048585+0000","last_clean":"2023-09-23T03:47:53.048585+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:47:53.048585+0000","last_undegraded":"2023-09-23T03:47:53.048585+0000","last_fullsized":"2023-09-23T03:47:53.048585+0000","mapping_epoch":18,"log_start":"149'217594","ondisk_log_start":"149'217594","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4791308307,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":227192,"num_read_kb":889760,"num_write":224719,"num_write_kb":898870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'9699","reported_seq":17593,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:50.606859+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:47:50.606859+0000","last_peered":"2023-09-23T03:47:50.606859+0000","last_clean":"2023-09-23T03:47:50.606859+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:47:50.606859+0000","last_undegraded":"2023-09-23T03:47:50.606859+0000","last_fullsized":"2023-09-23T03:47:50.606859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":320,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.792418+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:28.792418+0000","last_peered":"2023-09-23T03:46:28.792418+0000","last_clean":"2023-09-23T03:46:28.792418+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:28.792418+0000","last_undegraded":"2023-09-23T03:46:28.792418+0000","last_fullsized":"2023-09-23T03:46:28.792418+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20297502739,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":962717,"num_read_kb":3768404,"num_write":951669,"num_write_kb":3806670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35157553152,"data_stored":98803326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40148,"ondisk_log_size":40148,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44052,"ondisk_log_size":44052,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739188,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13032204,"kb_used_data":11444868,"kb_used_omap":1243,"kb_used_meta":1586084,"kb_avail":81339636,"statfs":{"total":96636764160,"available":83291787264,"internally_reserved":0,"allocated":11719544832,"data_stored":32935098225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1624150188},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,10,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1,"15min":1.0349999999999999},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.6560000000000001,"5min":6.6609999999999996,"15min":22.789000000000001},"last":0.221},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53100000000000003,"15min":0.55600000000000005},"min":{"1min":0.21299999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.3639999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.25800000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.71799999999999997,"15min":0.58099999999999996},"min":{"1min":0.23300000000000001,"5min":0.214,"15min":0.191},"max":{"1min":1.8700000000000001,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.94799999999999995,"15min":0.83599999999999997},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":2.1589999999999998,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.316}]}]},{"osd":1,"up_from":8,"seq":34359739187,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13019360,"kb_used_data":11447768,"kb_used_omap":1255,"kb_used_meta":1570328,"kb_avail":81352480,"statfs":{"total":96636764160,"available":83304939520,"internally_reserved":0,"allocated":11722514432,"data_stored":32935734435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1608016721},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,14,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.97499999999999998,"15min":0.995},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":5.9029999999999996,"5min":7.0410000000000004,"15min":10.444000000000001},"last":2.8100000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.78400000000000003,"15min":0.84199999999999997},"min":{"1min":0.22900000000000001,"5min":0.20300000000000001,"15min":0.185},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":10.307},"last":0.245}]},{"osd":2,"last update":"Sat Sep 23 03:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.72899999999999998,"15min":0.71199999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":3.327,"5min":4.8129999999999997,"15min":9.7080000000000002},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.69299999999999995,"15min":0.67500000000000004},"min":{"1min":0.25900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":0.52200000000000002,"5min":5.5960000000000001,"15min":10.313000000000001},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739184,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13018592,"kb_used_data":11447380,"kb_used_omap":1250,"kb_used_meta":1569949,"kb_avail":81353248,"statfs":{"total":96636764160,"available":83305725952,"internally_reserved":0,"allocated":11722117120,"data_stored":32935725012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1607628625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.55600000000000005,"15min":0.754},"min":{"1min":0.24099999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":0.68600000000000005,"5min":3.988,"15min":11.715},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.73299999999999998,"15min":0.95299999999999996},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":4.2830000000000004,"5min":5.9349999999999996,"15min":12.058},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.44,"15min":0.48299999999999998},"min":{"1min":0.27200000000000002,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":1.331,"5min":1.591,"15min":4.9720000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.46899999999999997,"15min":0.52900000000000003},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":4.8929999999999998,"5min":4.8929999999999998,"15min":8.6750000000000007},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11719520256,"data_stored":32934428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11720060928,"data_stored":32934477824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11717971968,"data_stored":32934420480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:47:54.367 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:54.367 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:54.367 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:54.367 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:47:54.367 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:48:14.369 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:48:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:48:15.153 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2206,"stamp":"2023-09-23T03:48:14.193621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20314790420,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1119927,"num_read_kb":3913078,"num_write":989989,"num_write_kb":3857629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84555,"ondisk_log_size":84555,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39345444,"kb_used_data":34380680,"kb_used_omap":3749,"kb_used_meta":4960986,"kb_avail":243770076,"statfs":{"total":289910292480,"available":249620557824,"internally_reserved":0,"allocated":35205816320,"data_stored":98807294952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":5080050510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,11,8,5,15,10,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12537856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3092,"num_read_kb":12324,"num_write":3008,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002130"},"pg_stats":[{"pgid":"2.7","version":"152'7686","reported_seq":12531,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:13.009358+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:48:13.009358+0000","last_peered":"2023-09-23T03:48:13.009358+0000","last_clean":"2023-09-23T03:48:13.009358+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:48:13.009358+0000","last_undegraded":"2023-09-23T03:48:13.009358+0000","last_fullsized":"2023-09-23T03:48:13.009358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7686,"log_dups_size":0,"ondisk_log_size":7686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'5268","reported_seq":9101,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:11.076262+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:48:11.076262+0000","last_peered":"2023-09-23T03:48:11.076262+0000","last_clean":"2023-09-23T03:48:11.076262+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:48:11.076262+0000","last_undegraded":"2023-09-23T03:48:11.076262+0000","last_fullsized":"2023-09-23T03:48:11.076262+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'6661","reported_seq":11457,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:13.003857+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:48:13.003857+0000","last_peered":"2023-09-23T03:48:13.003857+0000","last_clean":"2023-09-23T03:48:13.003857+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:48:13.003857+0000","last_undegraded":"2023-09-23T03:48:13.003857+0000","last_fullsized":"2023-09-23T03:48:13.003857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'9236","reported_seq":16816,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:13.008991+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:48:13.008991+0000","last_peered":"2023-09-23T03:48:13.008991+0000","last_clean":"2023-09-23T03:48:13.008991+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:48:13.008991+0000","last_undegraded":"2023-09-23T03:48:13.008991+0000","last_fullsized":"2023-09-23T03:48:13.008991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'241063","reported_seq":479217,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:12.465514+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:48:12.465514+0000","last_peered":"2023-09-23T03:48:12.465514+0000","last_clean":"2023-09-23T03:48:12.465514+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:48:12.465514+0000","last_undegraded":"2023-09-23T03:48:12.465514+0000","last_fullsized":"2023-09-23T03:48:12.465514+0000","mapping_epoch":18,"log_start":"149'230968","ondisk_log_start":"149'230968","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1548,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 122s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5063159808,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":240801,"num_read_kb":942380,"num_write":238039,"num_write_kb":952156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2251,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:56.602443+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:56.602443+0000","last_peered":"2023-09-23T03:46:56.602443+0000","last_clean":"2023-09-23T03:46:56.602443+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:56.602443+0000","last_undegraded":"2023-09-23T03:46:56.602443+0000","last_fullsized":"2023-09-23T03:46:56.602443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'252116","reported_seq":501366,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:13.366240+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:48:13.366240+0000","last_peered":"2023-09-23T03:48:13.366240+0000","last_clean":"2023-09-23T03:48:13.366240+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:48:13.366240+0000","last_undegraded":"2023-09-23T03:48:13.366240+0000","last_fullsized":"2023-09-23T03:48:13.366240+0000","mapping_epoch":18,"log_start":"149'242081","ondisk_log_start":"149'242081","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":822,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 106s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5294579712,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":252092,"num_read_kb":986236,"num_write":248935,"num_write_kb":995740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5642,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:05.961657+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:48:05.961657+0000","last_peered":"2023-09-23T03:48:05.961657+0000","last_clean":"2023-09-23T03:48:05.961657+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:48:05.961657+0000","last_undegraded":"2023-09-23T03:48:05.961657+0000","last_fullsized":"2023-09-23T03:48:05.961657+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'246122","reported_seq":489158,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:12.445997+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:48:12.445997+0000","last_peered":"2023-09-23T03:48:12.445997+0000","last_clean":"2023-09-23T03:48:12.445997+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:48:12.445997+0000","last_undegraded":"2023-09-23T03:48:12.445997+0000","last_fullsized":"2023-09-23T03:48:12.445997+0000","mapping_epoch":18,"log_start":"149'236024","ondisk_log_start":"149'236024","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1154,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 116s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5161250816,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":245762,"num_read_kb":962500,"num_write":243038,"num_write_kb":972152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4826,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:05.962452+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:48:05.962452+0000","last_peered":"2023-09-23T03:48:05.962452+0000","last_clean":"2023-09-23T03:48:05.962452+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:48:05.962452+0000","last_undegraded":"2023-09-23T03:48:05.962452+0000","last_fullsized":"2023-09-23T03:48:05.962452+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'228582","reported_seq":454105,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:13.368969+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:48:13.368969+0000","last_peered":"2023-09-23T03:48:13.368969+0000","last_clean":"2023-09-23T03:48:13.368969+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:48:13.368969+0000","last_undegraded":"2023-09-23T03:48:13.368969+0000","last_fullsized":"2023-09-23T03:48:13.368969+0000","mapping_epoch":18,"log_start":"149'218502","ondisk_log_start":"149'218502","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4795195411,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":228157,"num_read_kb":893612,"num_write":225671,"num_write_kb":902678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'9729","reported_seq":17638,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:13.069400+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:48:13.069400+0000","last_peered":"2023-09-23T03:48:13.069400+0000","last_clean":"2023-09-23T03:48:13.069400+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:48:13.069400+0000","last_undegraded":"2023-09-23T03:48:13.069400+0000","last_fullsized":"2023-09-23T03:48:13.069400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":320,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:28.792418+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:46:28.792418+0000","last_peered":"2023-09-23T03:46:28.792418+0000","last_clean":"2023-09-23T03:46:28.792418+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:46:28.792418+0000","last_undegraded":"2023-09-23T03:46:28.792418+0000","last_fullsized":"2023-09-23T03:46:28.792418+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20314185747,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":966812,"num_read_kb":3784728,"num_write":955683,"num_write_kb":3822726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35199139840,"data_stored":98804121600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40308,"ondisk_log_size":40308,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44148,"ondisk_log_size":44148,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739192,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13135720,"kb_used_data":11458144,"kb_used_omap":1243,"kb_used_meta":1676324,"kb_avail":81236120,"statfs":{"total":96636764160,"available":83185786880,"internally_reserved":0,"allocated":11733139456,"data_stored":32935339889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1716555948},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.0049999999999999,"15min":1.0409999999999999},"min":{"1min":0.20599999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.181,"5min":6.6609999999999996,"15min":22.789000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.56699999999999995},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.3159999999999998,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.28499999999999998}]},{"osd":1,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.245,"5min":0.214,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.873,"15min":0.82499999999999996},"min":{"1min":0.29399999999999998,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":4.1020000000000003,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":5.0499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739191,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13087456,"kb_used_data":11461720,"kb_used_omap":1255,"kb_used_meta":1624472,"kb_avail":81284384,"statfs":{"total":96636764160,"available":83235209216,"internally_reserved":0,"allocated":11736801280,"data_stored":32935984291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1663460177},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,2,8,7,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0189999999999999,"15min":1.0029999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":6.093,"5min":7.0410000000000004,"15min":10.444000000000001},"last":1.22},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77500000000000002,"15min":0.81599999999999995},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":3.335,"5min":5.5890000000000004,"15min":8.984},"last":1.0720000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.89400000000000002,"15min":0.76000000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.76800000000000002,"15min":0.71999999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.734,"5min":6.734,"15min":10.313000000000001},"last":0.39100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13122268,"kb_used_data":11460816,"kb_used_omap":1250,"kb_used_meta":1660189,"kb_avail":81249572,"statfs":{"total":96636764160,"available":83199561728,"internally_reserved":0,"allocated":11735875584,"data_stored":32935970772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1700034385},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,9,1,1,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.73899999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":4.6689999999999996,"5min":4.6689999999999996,"15min":11.715},"last":5.6319999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.69399999999999995,"15min":0.92600000000000005},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":4.742,"5min":5.9349999999999996,"15min":12.058},"last":0.252}]},{"osd":2,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.95799999999999996,"5min":1.591,"15min":4.9720000000000004},"last":5.6820000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.49399999999999999,"15min":0.50900000000000001},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":2.8439999999999999,"5min":4.8929999999999998,"15min":5.0119999999999996},"last":5.7110000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11733397504,"data_stored":32934694912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11733966848,"data_stored":32934739968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11731775488,"data_stored":32934686720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:48:15.201 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:15.201 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:15.201 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:15.201 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:15.202 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:48:35.205 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:48:36.056 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:48:36.056 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:48:36.116 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2216,"stamp":"2023-09-23T03:48:34.197865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20331338260,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1124017,"num_read_kb":3929378,"num_write":993974,"num_write_kb":3873569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84568,"ondisk_log_size":84568,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39571072,"kb_used_data":34421736,"kb_used_omap":3749,"kb_used_meta":5145562,"kb_avail":243544448,"statfs":{"total":289910292480,"available":249389514752,"internally_reserved":0,"allocated":35247857664,"data_stored":98807794664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":5269056334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,3,9,7,7,14,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12562432,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3089,"num_read_kb":12308,"num_write":3018,"num_write_kb":12072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002150"},"pg_stats":[{"pgid":"2.7","version":"152'7701","reported_seq":12558,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:32.058104+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:48:32.058104+0000","last_peered":"2023-09-23T03:48:32.058104+0000","last_clean":"2023-09-23T03:48:32.058104+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:48:32.058104+0000","last_undegraded":"2023-09-23T03:48:32.058104+0000","last_fullsized":"2023-09-23T03:48:32.058104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'5273","reported_seq":9110,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:31.561048+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:48:31.561048+0000","last_peered":"2023-09-23T03:48:31.561048+0000","last_clean":"2023-09-23T03:48:31.561048+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:48:31.561048+0000","last_undegraded":"2023-09-23T03:48:31.561048+0000","last_fullsized":"2023-09-23T03:48:31.561048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'6676","reported_seq":11484,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:32.053145+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:48:32.053145+0000","last_peered":"2023-09-23T03:48:32.053145+0000","last_clean":"2023-09-23T03:48:32.053145+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:48:32.053145+0000","last_undegraded":"2023-09-23T03:48:32.053145+0000","last_fullsized":"2023-09-23T03:48:32.053145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'9256","reported_seq":16852,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:32.079250+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:48:32.079250+0000","last_peered":"2023-09-23T03:48:32.079250+0000","last_clean":"2023-09-23T03:48:32.079250+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:48:32.079250+0000","last_undegraded":"2023-09-23T03:48:32.079250+0000","last_fullsized":"2023-09-23T03:48:32.079250+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"152'242059","reported_seq":481259,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:32.736111+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:48:32.736111+0000","last_peered":"2023-09-23T03:48:32.736111+0000","last_clean":"2023-09-23T03:48:32.736111+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:48:32.736111+0000","last_undegraded":"2023-09-23T03:48:32.736111+0000","last_fullsized":"2023-09-23T03:48:32.736111+0000","mapping_epoch":18,"log_start":"149'231980","ondisk_log_start":"149'231980","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1792,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5067337728,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":241831,"num_read_kb":946476,"num_write":239035,"num_write_kb":956140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2251,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-09-23T03:46:56.602443+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:46:56.602443+0000","last_peered":"2023-09-23T03:46:56.602443+0000","last_clean":"2023-09-23T03:46:56.602443+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:46:56.602443+0000","last_undegraded":"2023-09-23T03:46:56.602443+0000","last_fullsized":"2023-09-23T03:46:56.602443+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"152'253166","reported_seq":503503,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:33.606269+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:48:33.606269+0000","last_peered":"2023-09-23T03:48:33.606269+0000","last_clean":"2023-09-23T03:48:33.606269+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:48:33.606269+0000","last_undegraded":"2023-09-23T03:48:33.606269+0000","last_fullsized":"2023-09-23T03:48:33.606269+0000","mapping_epoch":18,"log_start":"150'243097","ondisk_log_start":"150'243097","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":977,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 126s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5298888704,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":253164,"num_read_kb":990512,"num_write":249985,"num_write_kb":999940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5650,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:26.441730+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:48:26.441730+0000","last_peered":"2023-09-23T03:48:26.441730+0000","last_clean":"2023-09-23T03:48:26.441730+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:48:26.441730+0000","last_undegraded":"2023-09-23T03:48:26.441730+0000","last_fullsized":"2023-09-23T03:48:26.441730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"152'247113","reported_seq":491202,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:32.730530+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:48:32.730530+0000","last_peered":"2023-09-23T03:48:32.730530+0000","last_clean":"2023-09-23T03:48:32.730530+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:48:32.730530+0000","last_undegraded":"2023-09-23T03:48:32.730530+0000","last_fullsized":"2023-09-23T03:48:32.730530+0000","mapping_epoch":18,"log_start":"150'237036","ondisk_log_start":"150'237036","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1414,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 136s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5165355008,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":246806,"num_read_kb":966668,"num_write":244029,"num_write_kb":976116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4835,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:26.442920+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:48:26.442920+0000","last_peered":"2023-09-23T03:48:26.442920+0000","last_clean":"2023-09-23T03:48:26.442920+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:48:26.442920+0000","last_undegraded":"2023-09-23T03:48:26.442920+0000","last_fullsized":"2023-09-23T03:48:26.442920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"152'229530","reported_seq":455999,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:33.574855+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:48:33.574855+0000","last_peered":"2023-09-23T03:48:33.574855+0000","last_clean":"2023-09-23T03:48:33.574855+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:48:33.574855+0000","last_undegraded":"2023-09-23T03:48:33.574855+0000","last_fullsized":"2023-09-23T03:48:33.574855+0000","mapping_epoch":18,"log_start":"149'219514","ondisk_log_start":"149'219514","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4799152147,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":229101,"num_read_kb":897372,"num_write":226619,"num_write_kb":906470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'9754","reported_seq":17683,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:32.197203+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:48:32.197203+0000","last_peered":"2023-09-23T03:48:32.197203+0000","last_clean":"2023-09-23T03:48:32.197203+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:48:32.197203+0000","last_undegraded":"2023-09-23T03:48:32.197203+0000","last_fullsized":"2023-09-23T03:48:32.197203+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":321,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:43.799057+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:47:43.799057+0000","last_peered":"2023-09-23T03:47:43.799057+0000","last_clean":"2023-09-23T03:47:43.799057+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:47:43.799057+0000","last_undegraded":"2023-09-23T03:47:43.799057+0000","last_fullsized":"2023-09-23T03:47:43.799057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20330733587,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":970902,"num_read_kb":3801028,"num_write":959668,"num_write_kb":3838666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35241078784,"data_stored":98804617216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40241,"ondisk_log_size":40241,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44228,"ondisk_log_size":44228,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13185720,"kb_used_data":11472048,"kb_used_omap":1243,"kb_used_meta":1712420,"kb_avail":81186120,"statfs":{"total":96636764160,"available":83134586880,"internally_reserved":0,"allocated":11747377152,"data_stored":32935507825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1753518252},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.0049999999999999,"15min":1.0409999999999999},"min":{"1min":0.20599999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.181,"5min":6.6609999999999996,"15min":22.789000000000001},"last":0.218},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.56699999999999995},"min":{"1min":0.23400000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.3159999999999998,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.315}]},{"osd":1,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.57799999999999996,"15min":0.56299999999999994},"min":{"1min":0.245,"5min":0.214,"15min":0.191},"max":{"1min":0.47699999999999998,"5min":7.0960000000000001,"15min":7.1509999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.873,"15min":0.82499999999999996},"min":{"1min":0.29399999999999998,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":4.1020000000000003,"5min":6.9000000000000004,"15min":7.1639999999999997},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359739195,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13191096,"kb_used_data":11475184,"kb_used_omap":1255,"kb_used_meta":1714648,"kb_avail":81180744,"statfs":{"total":96636764160,"available":83129081856,"internally_reserved":0,"allocated":11750588416,"data_stored":32936168611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1755800401},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,3,1,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0189999999999999,"15min":1.0029999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":6.093,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77500000000000002,"15min":0.81599999999999995},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":3.335,"5min":5.5890000000000004,"15min":8.984},"last":1.399}]},{"osd":2,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.89400000000000002,"15min":0.76000000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.76800000000000002,"15min":0.71999999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.734,"5min":6.734,"15min":10.313000000000001},"last":0.38600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739192,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13194256,"kb_used_data":11474504,"kb_used_omap":1250,"kb_used_meta":1718493,"kb_avail":81177584,"statfs":{"total":96636764160,"available":83125846016,"internally_reserved":0,"allocated":11749892096,"data_stored":32936118228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1759737681},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,6,3,4,6,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51100000000000001,"15min":0.73899999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":4.6689999999999996,"5min":4.6689999999999996,"15min":11.715},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.69399999999999995,"15min":0.92600000000000005},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":4.742,"5min":5.9349999999999996,"15min":12.058},"last":6.319}]},{"osd":2,"last update":"Sat Sep 23 03:47:54 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":0.95799999999999996,"5min":1.591,"15min":4.9720000000000004},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.49399999999999999,"15min":0.50900000000000001},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":2.8439999999999999,"5min":4.8929999999999998,"15min":5.0119999999999996},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11747422208,"data_stored":32934879232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11748077568,"data_stored":32934887424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11745579008,"data_stored":32934850560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:48:36.118 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:36.118 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:36.118 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:36.118 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:48:36.119 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-09-23T03:48:36.893 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-09-23T03:48:36.967 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-09-23T03:48:37.684 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-09-23T03:48:37.733 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-09-23T03:48:38.553 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-09-23T03:48:38.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-09-23T03:48:38.940 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.942+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:48:38.964 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.962+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:48:38.975 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:48:38.976 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:48:38.976 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:48:38.976 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:48:38.977 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:48:38.978 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:48:38.978 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:48:38.978 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:38.974+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:48:39.486 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-09-23T03:48:39.534 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:48:59.535 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:48:59.617 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:59.589+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:48:59.619 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:59.597+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:48:59.620 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:59.601+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:48:59.620 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:48:59.601+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:49:00.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:49:00.252 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:49:00.325 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2230,"stamp":"2023-09-23T03:49:00.202447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20351326740,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1128938,"num_read_kb":3948974,"num_write":998838,"num_write_kb":3893025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84627,"ondisk_log_size":84627,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39922256,"kb_used_data":34471796,"kb_used_omap":3749,"kb_used_meta":5446682,"kb_avail":243193264,"statfs":{"total":289910292480,"available":249029902336,"internally_reserved":0,"allocated":35299119104,"data_stored":98808195417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":5577403214},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,12,4,6,10,19,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7999488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1957,"num_read_kb":7788,"num_write":1946,"num_write_kb":7784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001658"},"pg_stats":[{"pgid":"2.7","version":"153'7731","reported_seq":12604,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:57.157725+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:48:57.157725+0000","last_peered":"2023-09-23T03:48:57.157725+0000","last_clean":"2023-09-23T03:48:57.157725+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:48:57.157725+0000","last_undegraded":"2023-09-23T03:48:57.157725+0000","last_fullsized":"2023-09-23T03:48:57.157725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'5283","reported_seq":9126,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:56.438192+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:48:56.438192+0000","last_peered":"2023-09-23T03:48:56.438192+0000","last_clean":"2023-09-23T03:48:56.438192+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:48:56.438192+0000","last_undegraded":"2023-09-23T03:48:56.438192+0000","last_fullsized":"2023-09-23T03:48:56.438192+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'6706","reported_seq":11530,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:57.157207+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:48:57.157207+0000","last_peered":"2023-09-23T03:48:57.157207+0000","last_clean":"2023-09-23T03:48:57.157207+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:48:57.157207+0000","last_undegraded":"2023-09-23T03:48:57.157207+0000","last_fullsized":"2023-09-23T03:48:57.157207+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'9296","reported_seq":16913,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:57.157267+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:48:57.157267+0000","last_peered":"2023-09-23T03:48:57.157267+0000","last_clean":"2023-09-23T03:48:57.157267+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:48:57.157267+0000","last_undegraded":"2023-09-23T03:48:57.157267+0000","last_fullsized":"2023-09-23T03:48:57.157267+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9296,"log_dups_size":0,"ondisk_log_size":9296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'243246","reported_seq":483665,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:57.959223+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:48:57.959223+0000","last_peered":"2023-09-23T03:48:57.959223+0000","last_clean":"2023-09-23T03:48:57.959223+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:48:57.959223+0000","last_undegraded":"2023-09-23T03:48:57.959223+0000","last_fullsized":"2023-09-23T03:48:57.959223+0000","mapping_epoch":18,"log_start":"150'233188","ondisk_log_start":"150'233188","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2157,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5072261120,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":243029,"num_read_kb":951236,"num_write":240222,"num_write_kb":960888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2252,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:47:43.798102+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:47:43.798102+0000","last_peered":"2023-09-23T03:47:43.798102+0000","last_clean":"2023-09-23T03:47:43.798102+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:47:43.798102+0000","last_undegraded":"2023-09-23T03:47:43.798102+0000","last_fullsized":"2023-09-23T03:47:43.798102+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'254448","reported_seq":506105,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:59.058391+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:48:59.058391+0000","last_peered":"2023-09-23T03:48:59.058391+0000","last_clean":"2023-09-23T03:48:59.058391+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:48:59.058391+0000","last_undegraded":"2023-09-23T03:48:59.058391+0000","last_fullsized":"2023-09-23T03:48:59.058391+0000","mapping_epoch":18,"log_start":"150'244425","ondisk_log_start":"150'244425","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1125,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 151s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5304168448,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":254467,"num_read_kb":995700,"num_write":251267,"num_write_kb":1005068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5662,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:52.041577+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:48:52.041577+0000","last_peered":"2023-09-23T03:48:52.041577+0000","last_clean":"2023-09-23T03:48:52.041577+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:48:52.041577+0000","last_undegraded":"2023-09-23T03:48:52.041577+0000","last_fullsized":"2023-09-23T03:48:52.041577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'248369","reported_seq":493739,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:48:58.019771+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:48:58.019771+0000","last_peered":"2023-09-23T03:48:58.019771+0000","last_clean":"2023-09-23T03:48:58.019771+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:48:58.019771+0000","last_undegraded":"2023-09-23T03:48:58.019771+0000","last_fullsized":"2023-09-23T03:48:58.019771+0000","mapping_epoch":18,"log_start":"150'238353","ondisk_log_start":"150'238353","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1672,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 161s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5170503680,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":248062,"num_read_kb":971676,"num_write":245285,"num_write_kb":981140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4846,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:52.065857+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:48:52.065857+0000","last_peered":"2023-09-23T03:48:52.065857+0000","last_clean":"2023-09-23T03:48:52.065857+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:48:52.065857+0000","last_undegraded":"2023-09-23T03:48:52.065857+0000","last_fullsized":"2023-09-23T03:48:52.065857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'230669","reported_seq":458304,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:59.054841+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:48:59.054841+0000","last_peered":"2023-09-23T03:48:59.054841+0000","last_clean":"2023-09-23T03:48:59.054841+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:48:59.054841+0000","last_undegraded":"2023-09-23T03:48:59.054841+0000","last_fullsized":"2023-09-23T03:48:59.054841+0000","mapping_epoch":18,"log_start":"150'220626","ondisk_log_start":"150'220626","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4803788819,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":230265,"num_read_kb":902012,"num_write":227758,"num_write_kb":911026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'9804","reported_seq":17759,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:57.164732+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:48:57.164732+0000","last_peered":"2023-09-23T03:48:57.164732+0000","last_clean":"2023-09-23T03:48:57.164732+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:48:57.164732+0000","last_undegraded":"2023-09-23T03:48:57.164732+0000","last_fullsized":"2023-09-23T03:48:57.164732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9804,"log_dups_size":0,"ondisk_log_size":9804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":322,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:23.507804+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:48:23.507804+0000","last_peered":"2023-09-23T03:48:23.507804+0000","last_clean":"2023-09-23T03:48:23.507804+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:48:23.507804+0000","last_undegraded":"2023-09-23T03:48:23.507804+0000","last_fullsized":"2023-09-23T03:48:23.507804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20350722067,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":975823,"num_read_kb":3820624,"num_write":964532,"num_write_kb":3858122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35292385280,"data_stored":98805035008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40140,"ondisk_log_size":40140,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44388,"ondisk_log_size":44388,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739201,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13314728,"kb_used_data":11488608,"kb_used_omap":1243,"kb_used_meta":1824868,"kb_avail":81057112,"statfs":{"total":96636764160,"available":83002482688,"internally_reserved":0,"allocated":11764334592,"data_stored":32935648236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1868665004},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,1,4,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.94699999999999995,"15min":1.0409999999999999},"min":{"1min":0.183,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":22.789000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":4.4359999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.26400000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.503,"15min":0.55400000000000005},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.191},"max":{"1min":0.56200000000000006,"5min":4.0499999999999998,"15min":7.1509999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.307,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.0499999999999998,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.51500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739200,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13302556,"kb_used_data":11492308,"kb_used_omap":1255,"kb_used_meta":1808984,"kb_avail":81069284,"statfs":{"total":96636764160,"available":83014946816,"internally_reserved":0,"allocated":11768123392,"data_stored":32936268062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1852400465},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,1,2,1,9,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.0189999999999999,"15min":1.0029999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":6.093,"5min":7.0410000000000004,"15min":10.444000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.77500000000000002,"15min":0.81599999999999995},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":3.335,"5min":5.5890000000000004,"15min":8.984},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:48:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":0.89400000000000002,"15min":0.76000000000000001},"min":{"1min":0.245,"5min":0.21099999999999999,"15min":0.20499999999999999},"max":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.76800000000000002,"15min":0.71999999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.734,"5min":6.734,"15min":10.313000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739197,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13304972,"kb_used_data":11490880,"kb_used_omap":1250,"kb_used_meta":1812829,"kb_avail":81066868,"statfs":{"total":96636764160,"available":83012472832,"internally_reserved":0,"allocated":11766661120,"data_stored":32936279119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1856337745},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,5,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.60299999999999998,"15min":0.75600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":11.715},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.071,"5min":0.72599999999999998,"15min":0.90300000000000002},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":6.319,"5min":6.319,"15min":12.058},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.505,"15min":0.505},"min":{"1min":0.251,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":5.6820000000000004,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.54900000000000004,"15min":0.52700000000000002},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11764523008,"data_stored":32935038976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11765256192,"data_stored":32935026688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11762606080,"data_stored":32934969344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:49:00.327 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:00.327 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:00.327 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:00.328 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:00.328 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:49:12.632 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.633+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:49:12.639 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:49:12.640 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.637+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:49:12.647 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.645+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:49:12.647 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.645+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:49:12.647 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:12.645+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:49:20.329 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:49:21.346 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:49:21.348 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:49:21.388 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2240,"stamp":"2023-09-23T03:49:20.209434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20367182356,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1132877,"num_read_kb":3964678,"num_write":1002690,"num_write_kb":3908433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84978,"ondisk_log_size":84978,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":40126328,"kb_used_data":34511444,"kb_used_omap":3749,"kb_used_meta":5611098,"kb_avail":242989192,"statfs":{"total":289910292480,"available":248820932608,"internally_reserved":0,"allocated":35339718656,"data_stored":98808617305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":5745765198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,10,23,11,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9867264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2498,"num_read_kb":9960,"num_write":2397,"num_write_kb":9588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002509"},"pg_stats":[{"pgid":"2.7","version":"153'7757","reported_seq":12642,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:17.636534+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:49:17.636534+0000","last_peered":"2023-09-23T03:49:17.636534+0000","last_clean":"2023-09-23T03:49:17.636534+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:49:17.636534+0000","last_undegraded":"2023-09-23T03:49:17.636534+0000","last_fullsized":"2023-09-23T03:49:17.636534+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'5292","reported_seq":9139,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:17.020908+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:49:17.020908+0000","last_peered":"2023-09-23T03:49:17.020908+0000","last_clean":"2023-09-23T03:49:17.020908+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:49:17.020908+0000","last_undegraded":"2023-09-23T03:49:17.020908+0000","last_fullsized":"2023-09-23T03:49:17.020908+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'6732","reported_seq":11569,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:17.636436+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:49:17.636436+0000","last_peered":"2023-09-23T03:49:17.636436+0000","last_clean":"2023-09-23T03:49:17.636436+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:49:17.636436+0000","last_undegraded":"2023-09-23T03:49:17.636436+0000","last_fullsized":"2023-09-23T03:49:17.636436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6732,"log_dups_size":0,"ondisk_log_size":6732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'9329","reported_seq":16963,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:17.636547+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:49:17.636547+0000","last_peered":"2023-09-23T03:49:17.636547+0000","last_clean":"2023-09-23T03:49:17.636547+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:49:17.636547+0000","last_undegraded":"2023-09-23T03:49:17.636547+0000","last_fullsized":"2023-09-23T03:49:17.636547+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9329,"log_dups_size":0,"ondisk_log_size":9329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'244194","reported_seq":485576,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:18.226529+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:49:18.226529+0000","last_peered":"2023-09-23T03:49:18.226529+0000","last_clean":"2023-09-23T03:49:18.226529+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:49:18.226529+0000","last_undegraded":"2023-09-23T03:49:18.226529+0000","last_fullsized":"2023-09-23T03:49:18.226529+0000","mapping_epoch":18,"log_start":"150'234105","ondisk_log_start":"150'234105","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2416,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 188s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5076123648,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":243978,"num_read_kb":955024,"num_write":241170,"num_write_kb":964680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2253,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:53.828572+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:48:53.828572+0000","last_peered":"2023-09-23T03:48:53.828572+0000","last_clean":"2023-09-23T03:48:53.828572+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:48:53.828572+0000","last_undegraded":"2023-09-23T03:48:53.828572+0000","last_fullsized":"2023-09-23T03:48:53.828572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'255426","reported_seq":508122,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:19.291854+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:49:19.291854+0000","last_peered":"2023-09-23T03:49:19.291854+0000","last_clean":"2023-09-23T03:49:19.291854+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:49:19.291854+0000","last_undegraded":"2023-09-23T03:49:19.291854+0000","last_fullsized":"2023-09-23T03:49:19.291854+0000","mapping_epoch":18,"log_start":"150'245334","ondisk_log_start":"150'245334","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1337,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 172s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5308166144,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":255491,"num_read_kb":999788,"num_write":252245,"num_write_kb":1008980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5670,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:12.543115+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:49:12.543115+0000","last_peered":"2023-09-23T03:49:12.543115+0000","last_clean":"2023-09-23T03:49:12.543115+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:49:12.543115+0000","last_undegraded":"2023-09-23T03:49:12.543115+0000","last_fullsized":"2023-09-23T03:49:12.543115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'249350","reported_seq":495746,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:18.235686+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:49:18.235686+0000","last_peered":"2023-09-23T03:49:18.235686+0000","last_clean":"2023-09-23T03:49:18.235686+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:49:18.235686+0000","last_undegraded":"2023-09-23T03:49:18.235686+0000","last_fullsized":"2023-09-23T03:49:18.235686+0000","mapping_epoch":18,"log_start":"150'239261","ondisk_log_start":"150'239261","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1852,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 182s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5174566912,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":249074,"num_read_kb":975700,"num_write":246266,"num_write_kb":985064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4854,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:12.520271+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:49:12.520271+0000","last_peered":"2023-09-23T03:49:12.520271+0000","last_clean":"2023-09-23T03:49:12.520271+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:49:12.520271+0000","last_undegraded":"2023-09-23T03:49:12.520271+0000","last_fullsized":"2023-09-23T03:49:12.520271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'231614","reported_seq":460204,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:19.307268+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:49:19.307268+0000","last_peered":"2023-09-23T03:49:19.307268+0000","last_clean":"2023-09-23T03:49:19.307268+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:49:19.307268+0000","last_undegraded":"2023-09-23T03:49:19.307268+0000","last_fullsized":"2023-09-23T03:49:19.307268+0000","mapping_epoch":18,"log_start":"150'221528","ondisk_log_start":"150'221528","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4807720979,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":231219,"num_read_kb":905816,"num_write":228703,"num_write_kb":914806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'9845","reported_seq":17821,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:17.636834+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:49:17.636834+0000","last_peered":"2023-09-23T03:49:17.636834+0000","last_clean":"2023-09-23T03:49:17.636834+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:49:17.636834+0000","last_undegraded":"2023-09-23T03:49:17.636834+0000","last_fullsized":"2023-09-23T03:49:17.636834+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":322,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:23.507804+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:48:23.507804+0000","last_peered":"2023-09-23T03:48:23.507804+0000","last_clean":"2023-09-23T03:48:23.507804+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:48:23.507804+0000","last_undegraded":"2023-09-23T03:48:23.507804+0000","last_fullsized":"2023-09-23T03:48:23.507804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20366577683,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":979762,"num_read_kb":3836328,"num_write":968384,"num_write_kb":3873530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35332706304,"data_stored":98805358592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40356,"ondisk_log_size":40356,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44523,"ondisk_log_size":44523,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13363852,"kb_used_data":11501632,"kb_used_omap":1243,"kb_used_meta":1860964,"kb_avail":81007988,"statfs":{"total":96636764160,"available":82952179712,"internally_reserved":0,"allocated":11777671168,"data_stored":32935807980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1905627308},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.94699999999999995,"15min":1.0409999999999999},"min":{"1min":0.183,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":22.789000000000001},"last":0.224},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":4.4359999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.49299999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.503,"15min":0.55400000000000005},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.191},"max":{"1min":0.56200000000000006,"5min":4.0499999999999998,"15min":7.1509999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.307,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.0499999999999998,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.33200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739204,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13388280,"kb_used_data":11505260,"kb_used_omap":1255,"kb_used_meta":1881752,"kb_avail":80983560,"statfs":{"total":96636764160,"available":82927165440,"internally_reserved":0,"allocated":11781386240,"data_stored":32936411422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1926914897},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,9,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94299999999999995,"15min":0.97799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":4.0940000000000003,"5min":6.093,"15min":10.444000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.74099999999999999,"15min":0.80300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":1.399,"5min":5.5890000000000004,"15min":8.984},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.89200000000000002,"15min":0.76000000000000001},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20499999999999999},"max":{"1min":0.68400000000000005,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.79200000000000004,"15min":0.73899999999999999},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":5.5469999999999997,"5min":6.734,"15min":10.313000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739201,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13374196,"kb_used_data":11504552,"kb_used_omap":1250,"kb_used_meta":1868381,"kb_avail":80997644,"statfs":{"total":96636764160,"available":82941587456,"internally_reserved":0,"allocated":11780661248,"data_stored":32936397903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":1913222993},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,12,2,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.60299999999999998,"15min":0.75600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":11.715},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.071,"5min":0.72599999999999998,"15min":0.90300000000000002},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":6.319,"5min":6.319,"15min":12.058},"last":0.67400000000000004}]},{"osd":2,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.505,"15min":0.505},"min":{"1min":0.251,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":5.6820000000000004,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.54900000000000004,"15min":0.52700000000000002},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.80000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11778002944,"data_stored":32935124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11778719744,"data_stored":32935112704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11775983616,"data_stored":32935120896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:49:21.390 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:21.390 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:21.391 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:21.391 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:21.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:49:22.174 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.160+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:49:22.188 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.184+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.200+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.200+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.200+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.200+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.200+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.204+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:49:22.207 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.204+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:49:22.208 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.204+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:49:22.208 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.204+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:49:22.208 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:49:22.204+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:49:41.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:49:42.158 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:49:42.159 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:49:42.204 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-09-23T03:49:40.212963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20383476244,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1136884,"num_read_kb":3980654,"num_write":1006659,"num_write_kb":3924309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84880,"ondisk_log_size":84880,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":40403760,"kb_used_data":34551256,"kb_used_omap":3749,"kb_used_meta":5848730,"kb_avail":242711760,"statfs":{"total":289910292480,"available":248536842240,"internally_reserved":0,"allocated":35380486144,"data_stored":98808854873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":5989100366},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,12,15,13,21,10,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12148736,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3040,"num_read_kb":12132,"num_write":2965,"num_write_kb":11860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002119"},"pg_stats":[{"pgid":"2.7","version":"153'7772","reported_seq":12669,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:38.116487+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:49:38.116487+0000","last_peered":"2023-09-23T03:49:38.116487+0000","last_clean":"2023-09-23T03:49:38.116487+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:49:38.116487+0000","last_undegraded":"2023-09-23T03:49:38.116487+0000","last_fullsized":"2023-09-23T03:49:38.116487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7772,"log_dups_size":0,"ondisk_log_size":7772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'5297","reported_seq":9148,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:35.688715+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:49:35.688715+0000","last_peered":"2023-09-23T03:49:35.688715+0000","last_clean":"2023-09-23T03:49:35.688715+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:49:35.688715+0000","last_undegraded":"2023-09-23T03:49:35.688715+0000","last_fullsized":"2023-09-23T03:49:35.688715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'6747","reported_seq":11596,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:38.116292+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:49:38.116292+0000","last_peered":"2023-09-23T03:49:38.116292+0000","last_clean":"2023-09-23T03:49:38.116292+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:49:38.116292+0000","last_undegraded":"2023-09-23T03:49:38.116292+0000","last_fullsized":"2023-09-23T03:49:38.116292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6747,"log_dups_size":0,"ondisk_log_size":6747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'9349","reported_seq":16999,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:38.117080+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:49:38.117080+0000","last_peered":"2023-09-23T03:49:38.117080+0000","last_clean":"2023-09-23T03:49:38.117080+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:49:38.117080+0000","last_undegraded":"2023-09-23T03:49:38.117080+0000","last_fullsized":"2023-09-23T03:49:38.117080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'245188","reported_seq":487615,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:38.404012+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:49:38.404012+0000","last_peered":"2023-09-23T03:49:38.404012+0000","last_clean":"2023-09-23T03:49:38.404012+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:49:38.404012+0000","last_undegraded":"2023-09-23T03:49:38.404012+0000","last_fullsized":"2023-09-23T03:49:38.404012+0000","mapping_epoch":18,"log_start":"150'235126","ondisk_log_start":"150'235126","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2602,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 208s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5080211456,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":245006,"num_read_kb":959124,"num_write":242164,"num_write_kb":968656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2253,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:53.828572+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:48:53.828572+0000","last_peered":"2023-09-23T03:48:53.828572+0000","last_clean":"2023-09-23T03:48:53.828572+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:48:53.828572+0000","last_undegraded":"2023-09-23T03:48:53.828572+0000","last_fullsized":"2023-09-23T03:48:53.828572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'256404","reported_seq":510110,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:39.601890+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:49:39.601890+0000","last_peered":"2023-09-23T03:49:39.601890+0000","last_clean":"2023-09-23T03:49:39.601890+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:49:39.601890+0000","last_undegraded":"2023-09-23T03:49:39.601890+0000","last_fullsized":"2023-09-23T03:49:39.601890+0000","mapping_epoch":18,"log_start":"150'246337","ondisk_log_start":"150'246337","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1536,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 192s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5312176128,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":256489,"num_read_kb":1003768,"num_write":253223,"num_write_kb":1012892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5678,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:33.005493+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:49:33.005493+0000","last_peered":"2023-09-23T03:49:33.005493+0000","last_clean":"2023-09-23T03:49:33.005493+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:49:33.005493+0000","last_undegraded":"2023-09-23T03:49:33.005493+0000","last_fullsized":"2023-09-23T03:49:33.005493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'250398","reported_seq":497847,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:38.396287+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:49:38.396287+0000","last_peered":"2023-09-23T03:49:38.396287+0000","last_clean":"2023-09-23T03:49:38.396287+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:49:38.396287+0000","last_undegraded":"2023-09-23T03:49:38.396287+0000","last_fullsized":"2023-09-23T03:49:38.396287+0000","mapping_epoch":18,"log_start":"150'240376","ondisk_log_start":"150'240376","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1982,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 202s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5178847232,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":250118,"num_read_kb":979852,"num_write":247314,"num_write_kb":989256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4862,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:33.015189+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:49:33.015189+0000","last_peered":"2023-09-23T03:49:33.015189+0000","last_clean":"2023-09-23T03:49:33.015189+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:49:33.015189+0000","last_undegraded":"2023-09-23T03:49:33.015189+0000","last_fullsized":"2023-09-23T03:49:33.015189+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'232563","reported_seq":462090,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:39.614011+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:49:39.614011+0000","last_peered":"2023-09-23T03:49:39.614011+0000","last_clean":"2023-09-23T03:49:39.614011+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:49:39.614011+0000","last_undegraded":"2023-09-23T03:49:39.614011+0000","last_fullsized":"2023-09-23T03:49:39.614011+0000","mapping_epoch":18,"log_start":"150'222536","ondisk_log_start":"150'222536","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4811636755,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":232156,"num_read_kb":909560,"num_write":229652,"num_write_kb":918602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'9870","reported_seq":17866,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:38.121152+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:49:38.121152+0000","last_peered":"2023-09-23T03:49:38.121152+0000","last_clean":"2023-09-23T03:49:38.121152+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:49:38.121152+0000","last_undegraded":"2023-09-23T03:49:38.121152+0000","last_fullsized":"2023-09-23T03:49:38.121152+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9870,"log_dups_size":0,"ondisk_log_size":9870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":322,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:23.507804+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:48:23.507804+0000","last_peered":"2023-09-23T03:48:23.507804+0000","last_clean":"2023-09-23T03:48:23.507804+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:48:23.507804+0000","last_undegraded":"2023-09-23T03:48:23.507804+0000","last_fullsized":"2023-09-23T03:48:23.507804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20382871571,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":983769,"num_read_kb":3852304,"num_write":972353,"num_write_kb":3889406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35374366720,"data_stored":98805608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40178,"ondisk_log_size":40178,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44603,"ondisk_log_size":44603,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13467844,"kb_used_data":11515004,"kb_used_omap":1243,"kb_used_meta":1951588,"kb_avail":80903996,"statfs":{"total":96636764160,"available":82845691904,"internally_reserved":0,"allocated":11791364096,"data_stored":32935861228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":1998426284},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,3,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.94699999999999995,"15min":1.0409999999999999},"min":{"1min":0.183,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":22.789000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53600000000000003,"15min":0.58099999999999996},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.19500000000000001},"max":{"1min":4.4359999999999999,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.26900000000000002}]},{"osd":1,"last update":"Sat Sep 23 03:48:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.503,"15min":0.55400000000000005},"min":{"1min":0.24399999999999999,"5min":0.214,"15min":0.191},"max":{"1min":0.56200000000000006,"5min":4.0499999999999998,"15min":7.1509999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.76400000000000001,"15min":0.80400000000000005},"min":{"1min":0.307,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.0499999999999998,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.43099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739208,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13460080,"kb_used_data":11518824,"kb_used_omap":1255,"kb_used_meta":1939992,"kb_avail":80911760,"statfs":{"total":96636764160,"available":82853642240,"internally_reserved":0,"allocated":11795275776,"data_stored":32936497438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":1986552657},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94299999999999995,"15min":0.97799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":4.0940000000000003,"5min":6.093,"15min":10.444000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.74099999999999999,"15min":0.80300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":1.399,"5min":5.5890000000000004,"15min":8.984},"last":0.26000000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.89200000000000002,"15min":0.76000000000000001},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20499999999999999},"max":{"1min":0.68400000000000005,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.79200000000000004,"15min":0.73899999999999999},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":5.5469999999999997,"5min":6.734,"15min":10.313000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359739205,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13475836,"kb_used_data":11517428,"kb_used_omap":1250,"kb_used_meta":1957149,"kb_avail":80896004,"statfs":{"total":96636764160,"available":82837508096,"internally_reserved":0,"allocated":11793846272,"data_stored":32936496207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":2004121425},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,9,8,9,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.60299999999999998,"15min":0.75600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":11.715},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.071,"5min":0.72599999999999998,"15min":0.90300000000000002},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":6.319,"5min":6.319,"15min":12.058},"last":0.23799999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.505,"15min":0.505},"min":{"1min":0.251,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":5.6820000000000004,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.54900000000000004,"15min":0.52700000000000002},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11791851520,"data_stored":32935206912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11792678912,"data_stored":32935198720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11789836288,"data_stored":32935202816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:49:42.205 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:42.206 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:42.206 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:42.206 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:49:42.206 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:50:02.207 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:50:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:50:02.833 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:50:02.922 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2261,"stamp":"2023-09-23T03:50:02.216983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20399286804,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1140821,"num_read_kb":3996330,"num_write":1010504,"num_write_kb":3939689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84974,"ondisk_log_size":84974,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":40657020,"kb_used_data":34594404,"kb_used_omap":3749,"kb_used_meta":6058842,"kb_avail":242458500,"statfs":{"total":289910292480,"available":248277504000,"internally_reserved":0,"allocated":35424669696,"data_stored":98809268569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3839154,"internal_metadata":6204255054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,35,7,3,17,24,14,7,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7933952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1987,"num_read_kb":7908,"num_write":1928,"num_write_kb":7712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002084"},"pg_stats":[{"pgid":"2.7","version":"153'7793","reported_seq":12703,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:59.566298+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:49:59.566298+0000","last_peered":"2023-09-23T03:49:59.566298+0000","last_clean":"2023-09-23T03:49:59.566298+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:49:59.566298+0000","last_undegraded":"2023-09-23T03:49:59.566298+0000","last_fullsized":"2023-09-23T03:49:59.566298+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7793,"log_dups_size":0,"ondisk_log_size":7793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'5304","reported_seq":9159,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:58.596451+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:49:58.596451+0000","last_peered":"2023-09-23T03:49:58.596451+0000","last_clean":"2023-09-23T03:49:58.596451+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:49:58.596451+0000","last_undegraded":"2023-09-23T03:49:58.596451+0000","last_fullsized":"2023-09-23T03:49:58.596451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'6768","reported_seq":11629,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:59.559637+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:49:59.559637+0000","last_peered":"2023-09-23T03:49:59.559637+0000","last_clean":"2023-09-23T03:49:59.559637+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:49:59.559637+0000","last_undegraded":"2023-09-23T03:49:59.559637+0000","last_fullsized":"2023-09-23T03:49:59.559637+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6768,"log_dups_size":0,"ondisk_log_size":6768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'9377","reported_seq":17043,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:59.564353+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:49:59.564353+0000","last_peered":"2023-09-23T03:49:59.564353+0000","last_clean":"2023-09-23T03:49:59.564353+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:49:59.564353+0000","last_undegraded":"2023-09-23T03:49:59.564353+0000","last_fullsized":"2023-09-23T03:49:59.564353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9377,"log_dups_size":0,"ondisk_log_size":9377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"153'246137","reported_seq":489567,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:58.681069+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:49:58.681069+0000","last_peered":"2023-09-23T03:49:58.681069+0000","last_clean":"2023-09-23T03:49:58.681069+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:49:58.681069+0000","last_undegraded":"2023-09-23T03:49:58.681069+0000","last_fullsized":"2023-09-23T03:49:58.681069+0000","mapping_epoch":18,"log_start":"151'236134","ondisk_log_start":"151'236134","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2931,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5084106752,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":245995,"num_read_kb":963060,"num_write":243113,"num_write_kb":972452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2253,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:53.828572+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:48:53.828572+0000","last_peered":"2023-09-23T03:48:53.828572+0000","last_clean":"2023-09-23T03:48:53.828572+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:48:53.828572+0000","last_undegraded":"2023-09-23T03:48:53.828572+0000","last_fullsized":"2023-09-23T03:48:53.828572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"153'257432","reported_seq":512155,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:59.896251+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:49:59.896251+0000","last_peered":"2023-09-23T03:49:59.896251+0000","last_clean":"2023-09-23T03:49:59.896251+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:49:59.896251+0000","last_undegraded":"2023-09-23T03:49:59.896251+0000","last_fullsized":"2023-09-23T03:49:59.896251+0000","mapping_epoch":18,"log_start":"151'247355","ondisk_log_start":"151'247355","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1672,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 212s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5316431872,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":257492,"num_read_kb":1007772,"num_write":254251,"num_write_kb":1017004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5688,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:58.603297+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:49:58.603297+0000","last_peered":"2023-09-23T03:49:58.603297+0000","last_clean":"2023-09-23T03:49:58.603297+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:49:58.603297+0000","last_undegraded":"2023-09-23T03:49:58.603297+0000","last_fullsized":"2023-09-23T03:49:58.603297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'251372","reported_seq":499882,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:49:58.667372+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:49:58.667372+0000","last_peered":"2023-09-23T03:49:58.667372+0000","last_clean":"2023-09-23T03:49:58.667372+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:49:58.667372+0000","last_undegraded":"2023-09-23T03:49:58.667372+0000","last_fullsized":"2023-09-23T03:49:58.667372+0000","mapping_epoch":18,"log_start":"151'241304","ondisk_log_start":"151'241304","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2164,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 222s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5182840832,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":251166,"num_read_kb":984024,"num_write":248288,"num_write_kb":993152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4872,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:58.600102+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:49:58.600102+0000","last_peered":"2023-09-23T03:49:58.600102+0000","last_clean":"2023-09-23T03:49:58.600102+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:49:58.600102+0000","last_undegraded":"2023-09-23T03:49:58.600102+0000","last_fullsized":"2023-09-23T03:49:58.600102+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"153'233457","reported_seq":463880,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:59.898580+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:49:59.898580+0000","last_peered":"2023-09-23T03:49:59.898580+0000","last_clean":"2023-09-23T03:49:59.898580+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:49:59.898580+0000","last_undegraded":"2023-09-23T03:49:59.898580+0000","last_fullsized":"2023-09-23T03:49:59.898580+0000","mapping_epoch":18,"log_start":"150'223445","ondisk_log_start":"150'223445","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4815302675,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":233053,"num_read_kb":913124,"num_write":230546,"num_write_kb":922178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"153'9905","reported_seq":17921,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:59.753082+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:49:59.753082+0000","last_peered":"2023-09-23T03:49:59.753082+0000","last_clean":"2023-09-23T03:49:59.753082+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:49:59.753082+0000","last_undegraded":"2023-09-23T03:49:59.753082+0000","last_fullsized":"2023-09-23T03:49:59.753082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":322,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-09-23T03:48:23.507804+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:48:23.507804+0000","last_peered":"2023-09-23T03:48:23.507804+0000","last_clean":"2023-09-23T03:48:23.507804+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:48:23.507804+0000","last_undegraded":"2023-09-23T03:48:23.507804+0000","last_fullsized":"2023-09-23T03:48:23.507804+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20398682131,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":987706,"num_read_kb":3867980,"num_write":976198,"num_write_kb":3904786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35418136576,"data_stored":98806087680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40160,"ondisk_log_size":40160,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44715,"ondisk_log_size":44715,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739214,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13548172,"kb_used_data":11531524,"kb_used_omap":1243,"kb_used_meta":2015396,"kb_avail":80823668,"statfs":{"total":96636764160,"available":82763436032,"internally_reserved":0,"allocated":11808280576,"data_stored":32936033260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1273684,"internal_metadata":2063765676},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.91200000000000003,"15min":1.0209999999999999},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.1539999999999999,"5min":5.9240000000000004,"15min":22.789000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":1.851,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":1.851}]},{"osd":1,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42899999999999999,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":0.628,"5min":3.867,"15min":7.1509999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.753,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.2009999999999996,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359739212,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13563744,"kb_used_data":11531736,"kb_used_omap":1255,"kb_used_meta":2030744,"kb_avail":80808096,"statfs":{"total":96636764160,"available":82747490304,"internally_reserved":0,"allocated":11808497664,"data_stored":32936620318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285295,"internal_metadata":2079482705},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,1,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94299999999999995,"15min":0.97799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":4.0940000000000003,"5min":6.093,"15min":10.444000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.74099999999999999,"15min":0.80300000000000005},"min":{"1min":0.23400000000000001,"5min":0.22900000000000001,"15min":0.185},"max":{"1min":1.399,"5min":5.5890000000000004,"15min":8.984},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:49:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.89200000000000002,"15min":0.76000000000000001},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.20499999999999999},"max":{"1min":0.68400000000000005,"5min":5.7830000000000004,"15min":9.7080000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.79200000000000004,"15min":0.73899999999999999},"min":{"1min":0.24299999999999999,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":5.5469999999999997,"5min":6.734,"15min":10.313000000000001},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739210,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13545104,"kb_used_data":11531144,"kb_used_omap":1250,"kb_used_meta":2012701,"kb_avail":80826736,"statfs":{"total":96636764160,"available":82766577664,"internally_reserved":0,"allocated":11807891456,"data_stored":32936614991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1280175,"internal_metadata":2061006673},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,35,3,0,10,15,5,7,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.60299999999999998,"15min":0.75600000000000001},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":8.4030000000000005,"5min":8.4030000000000005,"15min":11.715},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.071,"5min":0.72599999999999998,"15min":0.90300000000000002},"min":{"1min":0.19600000000000001,"5min":0.183,"15min":0.183},"max":{"1min":6.319,"5min":6.319,"15min":12.058},"last":0.223}]},{"osd":2,"last update":"Sat Sep 23 03:48:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.505,"15min":0.505},"min":{"1min":0.251,"5min":0.25,"15min":0.20100000000000001},"max":{"1min":5.6820000000000004,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.54900000000000004,"15min":0.52700000000000002},"min":{"1min":0.26200000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":5.7110000000000003,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11805290496,"data_stored":32935346176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11806154752,"data_stored":32935391232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11806691328,"data_stored":32935350272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:50:02.923 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:02.924 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:02.924 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:02.924 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:02.924 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:50:15.124 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:50:15.124 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:50:15.125 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:50:15.126 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:50:15.126 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:50:15.126 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:50:15.127 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:50:15.127 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:50:15.127 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:15.119+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:50:22.926 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:50:23.654 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:50:23.654 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:50:23.703 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2272,"stamp":"2023-09-23T03:50:22.228705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20414818836,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1144561,"num_read_kb":4011226,"num_write":1014280,"num_write_kb":3954793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85191,"ondisk_log_size":85191,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39038792,"kb_used_data":34633392,"kb_used_omap":3754,"kb_used_meta":4401621,"kb_avail":244076728,"statfs":{"total":289910292480,"available":249934569472,"internally_reserved":0,"allocated":35464593408,"data_stored":98809701961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4507260831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,10,6,15,24,5,12,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7598080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1849,"num_read_kb":7356,"num_write":1858,"num_write_kb":7432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007655"},"pg_stats":[{"pgid":"2.7","version":"154'7829","reported_seq":12751,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.748091+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:50:19.748091+0000","last_peered":"2023-09-23T03:50:19.748091+0000","last_clean":"2023-09-23T03:50:19.748091+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:50:19.748091+0000","last_undegraded":"2023-09-23T03:50:19.748091+0000","last_fullsized":"2023-09-23T03:50:19.748091+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7829,"log_dups_size":0,"ondisk_log_size":7829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'5316","reported_seq":9177,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.263437+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:50:19.263437+0000","last_peered":"2023-09-23T03:50:19.263437+0000","last_clean":"2023-09-23T03:50:19.263437+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:50:19.263437+0000","last_undegraded":"2023-09-23T03:50:19.263437+0000","last_fullsized":"2023-09-23T03:50:19.263437+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'6804","reported_seq":11677,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.763709+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:50:19.763709+0000","last_peered":"2023-09-23T03:50:19.763709+0000","last_clean":"2023-09-23T03:50:19.763709+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:50:19.763709+0000","last_undegraded":"2023-09-23T03:50:19.763709+0000","last_fullsized":"2023-09-23T03:50:19.763709+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'9425","reported_seq":17105,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.774162+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:50:19.774162+0000","last_peered":"2023-09-23T03:50:19.774162+0000","last_clean":"2023-09-23T03:50:19.774162+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:50:19.774162+0000","last_undegraded":"2023-09-23T03:50:19.774162+0000","last_fullsized":"2023-09-23T03:50:19.774162+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9425,"log_dups_size":0,"ondisk_log_size":9425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'247075","reported_seq":491435,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:18.836897+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:50:18.836897+0000","last_peered":"2023-09-23T03:50:18.836897+0000","last_clean":"2023-09-23T03:50:18.836897+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:50:18.836897+0000","last_undegraded":"2023-09-23T03:50:18.836897+0000","last_fullsized":"2023-09-23T03:50:18.836897+0000","mapping_epoch":18,"log_start":"151'237052","ondisk_log_start":"151'237052","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3241,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 248s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5088043008,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":246903,"num_read_kb":966680,"num_write":244051,"num_write_kb":976204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2254,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:03.044271+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:49:03.044271+0000","last_peered":"2023-09-23T03:49:03.044271+0000","last_clean":"2023-09-23T03:49:03.044271+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:49:03.044271+0000","last_undegraded":"2023-09-23T03:49:03.044271+0000","last_fullsized":"2023-09-23T03:49:03.044271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'258454","reported_seq":514162,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:20.149369+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:50:20.149369+0000","last_peered":"2023-09-23T03:50:20.149369+0000","last_clean":"2023-09-23T03:50:20.149369+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:50:20.149369+0000","last_undegraded":"2023-09-23T03:50:20.149369+0000","last_fullsized":"2023-09-23T03:50:20.149369+0000","mapping_epoch":18,"log_start":"151'248373","ondisk_log_start":"151'248373","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1820,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 232s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5320597504,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":258464,"num_read_kb":1011640,"num_write":255273,"num_write_kb":1021092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5697,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.080387+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:50:19.080387+0000","last_peered":"2023-09-23T03:50:19.080387+0000","last_clean":"2023-09-23T03:50:19.080387+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:50:19.080387+0000","last_undegraded":"2023-09-23T03:50:19.080387+0000","last_fullsized":"2023-09-23T03:50:19.080387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'252347","reported_seq":501846,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:18.838125+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:50:18.838125+0000","last_peered":"2023-09-23T03:50:18.838125+0000","last_clean":"2023-09-23T03:50:18.838125+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:50:18.838125+0000","last_undegraded":"2023-09-23T03:50:18.838125+0000","last_fullsized":"2023-09-23T03:50:18.838125+0000","mapping_epoch":18,"log_start":"151'242311","ondisk_log_start":"151'242311","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2358,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 242s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5186842624,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":252138,"num_read_kb":987900,"num_write":249263,"num_write_kb":997052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4882,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.080384+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:50:19.080384+0000","last_peered":"2023-09-23T03:50:19.080384+0000","last_clean":"2023-09-23T03:50:19.080384+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:50:19.080384+0000","last_undegraded":"2023-09-23T03:50:19.080384+0000","last_fullsized":"2023-09-23T03:50:19.080384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'234298","reported_seq":465610,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:20.160652+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:50:20.160652+0000","last_peered":"2023-09-23T03:50:20.160652+0000","last_clean":"2023-09-23T03:50:20.160652+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:50:20.160652+0000","last_undegraded":"2023-09-23T03:50:20.160652+0000","last_fullsized":"2023-09-23T03:50:20.160652+0000","mapping_epoch":18,"log_start":"151'224251","ondisk_log_start":"151'224251","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4818731027,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":233941,"num_read_kb":916656,"num_write":231387,"num_write_kb":925542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'9963","reported_seq":17995,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:19.833921+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:50:19.833921+0000","last_peered":"2023-09-23T03:50:19.833921+0000","last_clean":"2023-09-23T03:50:19.833921+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:50:19.833921+0000","last_undegraded":"2023-09-23T03:50:19.833921+0000","last_fullsized":"2023-09-23T03:50:19.833921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":323,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:48:53.822019+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:48:53.822019+0000","last_peered":"2023-09-23T03:48:53.822019+0000","last_clean":"2023-09-23T03:48:53.822019+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:48:53.822019+0000","last_undegraded":"2023-09-23T03:48:53.822019+0000","last_fullsized":"2023-09-23T03:48:53.822019+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20414214163,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":991446,"num_read_kb":3882876,"num_write":979974,"num_write_kb":3919890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35458281472,"data_stored":98806538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":44905,"ondisk_log_size":44905,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12750056,"kb_used_data":11544416,"kb_used_omap":1245,"kb_used_meta":1204386,"kb_avail":81621784,"statfs":{"total":96636764160,"available":83580706816,"internally_reserved":0,"allocated":11821481984,"data_stored":32936173628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1233291804},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.91200000000000003,"15min":1.0209999999999999},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.1539999999999999,"5min":5.9240000000000004,"15min":22.789000000000001},"last":2.198},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":1.851,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42899999999999999,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":0.628,"5min":3.867,"15min":7.1509999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.753,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.2009999999999996,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12964680,"kb_used_data":11544832,"kb_used_omap":1256,"kb_used_meta":1418583,"kb_avail":81407160,"statfs":{"total":96636764160,"available":83360931840,"internally_reserved":0,"allocated":11821907968,"data_stored":32936756590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1452629186},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,8,14,4,12,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.27400000000000002,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":2.895,"5min":6.093,"15min":10.444000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85399999999999998,"15min":0.71299999999999997},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.806},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.72199999999999998},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.0940000000000003,"5min":6.734,"15min":10.313000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13324056,"kb_used_data":11544144,"kb_used_omap":1251,"kb_used_meta":1778652,"kb_avail":81047784,"statfs":{"total":96636764160,"available":82992930816,"internally_reserved":0,"allocated":11821203456,"data_stored":32936771743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1821339841},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,7,1,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.58699999999999997,"15min":0.754},"min":{"1min":0.25800000000000001,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":0.72299999999999998,"5min":8.4030000000000005,"15min":11.715},"last":0.253},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.72999999999999998,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.183,"15min":0.183},"max":{"1min":4.2469999999999999,"5min":6.319,"15min":12.058},"last":5.6189999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":1.522,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55800000000000005,"15min":0.53200000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":1.018,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11818688512,"data_stored":32935518208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11819577344,"data_stored":32935538688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11820015616,"data_stored":32935481344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:50:23.705 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:23.705 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:23.705 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:23.705 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:23.705 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:50:28.338 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.339+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:50:28.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:50:28.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:50:28.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:50:28.354 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:50:28.355 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:50:28.355 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.355+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:50:28.357 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.359+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:50:28.358 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.359+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:50:28.358 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.359+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:50:28.358 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:28.359+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:50:37.546 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:50:37.547 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:50:37.548 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:50:37.548 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:37.547+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:50:43.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:50:44.500 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:50:44.501 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:50:44.558 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2283,"stamp":"2023-09-23T03:50:44.232716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20433668628,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1148842,"num_read_kb":4028286,"num_write":1018801,"num_write_kb":3972877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85359,"ondisk_log_size":85359,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39359176,"kb_used_data":34678448,"kb_used_omap":3754,"kb_used_meta":4676949,"kb_avail":243756344,"statfs":{"total":289910292480,"available":249606496256,"internally_reserved":0,"allocated":35510730752,"data_stored":98810709577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4789196703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,12,18,44,17,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10498048,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2381,"num_read_kb":9504,"num_write":2492,"num_write_kb":9968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"2.7","version":"154'7850","reported_seq":12784,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.558956+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:50:39.558956+0000","last_peered":"2023-09-23T03:50:39.558956+0000","last_clean":"2023-09-23T03:50:39.558956+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:50:39.558956+0000","last_undegraded":"2023-09-23T03:50:39.558956+0000","last_fullsized":"2023-09-23T03:50:39.558956+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'5323","reported_seq":9188,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:37.540042+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:50:37.540042+0000","last_peered":"2023-09-23T03:50:37.540042+0000","last_clean":"2023-09-23T03:50:37.540042+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:50:37.540042+0000","last_undegraded":"2023-09-23T03:50:37.540042+0000","last_fullsized":"2023-09-23T03:50:37.540042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'6825","reported_seq":11710,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.556355+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:50:39.556355+0000","last_peered":"2023-09-23T03:50:39.556355+0000","last_clean":"2023-09-23T03:50:39.556355+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:50:39.556355+0000","last_undegraded":"2023-09-23T03:50:39.556355+0000","last_fullsized":"2023-09-23T03:50:39.556355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'9453","reported_seq":17149,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.556715+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:50:39.556715+0000","last_peered":"2023-09-23T03:50:39.556715+0000","last_clean":"2023-09-23T03:50:39.556715+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:50:39.556715+0000","last_undegraded":"2023-09-23T03:50:39.556715+0000","last_fullsized":"2023-09-23T03:50:39.556715+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9453,"log_dups_size":0,"ondisk_log_size":9453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'248270","reported_seq":493872,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:44.143694+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:50:44.143694+0000","last_peered":"2023-09-23T03:50:44.143694+0000","last_clean":"2023-09-23T03:50:44.143694+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:50:44.143694+0000","last_undegraded":"2023-09-23T03:50:44.143694+0000","last_fullsized":"2023-09-23T03:50:44.143694+0000","mapping_epoch":18,"log_start":"151'238171","ondisk_log_start":"151'238171","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3596,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 274s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5092937728,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":248126,"num_read_kb":971552,"num_write":245246,"num_write_kb":980984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2254,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:03.044271+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:49:03.044271+0000","last_peered":"2023-09-23T03:49:03.044271+0000","last_clean":"2023-09-23T03:49:03.044271+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:49:03.044271+0000","last_undegraded":"2023-09-23T03:49:03.044271+0000","last_fullsized":"2023-09-23T03:49:03.044271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'259475","reported_seq":516152,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:40.404116+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:50:40.404116+0000","last_peered":"2023-09-23T03:50:40.404116+0000","last_clean":"2023-09-23T03:50:40.404116+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:50:40.404116+0000","last_undegraded":"2023-09-23T03:50:40.404116+0000","last_fullsized":"2023-09-23T03:50:40.404116+0000","mapping_epoch":18,"log_start":"151'249382","ondisk_log_start":"151'249382","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2045,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5324972032,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":259420,"num_read_kb":1015456,"num_write":256294,"num_write_kb":1025176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5705,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.571892+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:50:39.571892+0000","last_peered":"2023-09-23T03:50:39.571892+0000","last_clean":"2023-09-23T03:50:39.571892+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:50:39.571892+0000","last_undegraded":"2023-09-23T03:50:39.571892+0000","last_fullsized":"2023-09-23T03:50:39.571892+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'253642","reported_seq":504388,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:50:44.128533+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:50:44.128533+0000","last_peered":"2023-09-23T03:50:44.128533+0000","last_clean":"2023-09-23T03:50:44.128533+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:50:44.128533+0000","last_undegraded":"2023-09-23T03:50:44.128533+0000","last_fullsized":"2023-09-23T03:50:44.128533+0000","mapping_epoch":18,"log_start":"151'243634","ondisk_log_start":"151'243634","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2587,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 268s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5192200192,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":253370,"num_read_kb":992808,"num_write":250558,"num_write_kb":1002232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4890,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.558139+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:50:39.558139+0000","last_peered":"2023-09-23T03:50:39.558139+0000","last_clean":"2023-09-23T03:50:39.558139+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:50:39.558139+0000","last_undegraded":"2023-09-23T03:50:39.558139+0000","last_fullsized":"2023-09-23T03:50:39.558139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'235308","reported_seq":467489,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:40.417433+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:50:40.417433+0000","last_peered":"2023-09-23T03:50:40.417433+0000","last_clean":"2023-09-23T03:50:40.417433+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:50:40.417433+0000","last_undegraded":"2023-09-23T03:50:40.417433+0000","last_fullsized":"2023-09-23T03:50:40.417433+0000","mapping_epoch":18,"log_start":"151'225265","ondisk_log_start":"151'225265","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4822954003,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":234811,"num_read_kb":920120,"num_write":232397,"num_write_kb":929582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'9998","reported_seq":18050,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:39.568429+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:50:39.568429+0000","last_peered":"2023-09-23T03:50:39.568429+0000","last_clean":"2023-09-23T03:50:39.568429+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:50:39.568429+0000","last_undegraded":"2023-09-23T03:50:39.568429+0000","last_fullsized":"2023-09-23T03:50:39.568429+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":9998,"log_dups_size":0,"ondisk_log_size":9998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":324,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:03.888844+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:50:03.888844+0000","last_peered":"2023-09-23T03:50:03.888844+0000","last_clean":"2023-09-23T03:50:03.888844+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:50:03.888844+0000","last_undegraded":"2023-09-23T03:50:03.888844+0000","last_fullsized":"2023-09-23T03:50:03.888844+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20433063955,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":995727,"num_read_kb":3899936,"num_write":984495,"num_write_kb":3937974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35504537600,"data_stored":98807480320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45017,"ondisk_log_size":45017,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739222,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12835936,"kb_used_data":11558168,"kb_used_omap":1245,"kb_used_meta":1276514,"kb_avail":81535904,"statfs":{"total":96636764160,"available":83492765696,"internally_reserved":0,"allocated":11835564032,"data_stored":32936468540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1307150876},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.91200000000000003,"15min":1.0209999999999999},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":1.1539999999999999,"5min":5.9240000000000004,"15min":22.789000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.55300000000000005,"15min":0.56799999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":1.851,"5min":4.9020000000000001,"15min":5.1779999999999999},"last":0.20200000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.42899999999999999,"15min":0.55700000000000005},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":0.628,"5min":3.867,"15min":7.1509999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.753,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.223,"15min":0.20999999999999999},"max":{"1min":5.2009999999999996,"5min":5.7990000000000004,"15min":7.1639999999999997},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359739220,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13087732,"kb_used_data":11559084,"kb_used_omap":1256,"kb_used_meta":1527383,"kb_avail":81284108,"statfs":{"total":96636764160,"available":83234926592,"internally_reserved":0,"allocated":11836502016,"data_stored":32937088366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1564040386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,9,3,20,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.27400000000000002,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":2.895,"5min":6.093,"15min":10.444000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85399999999999998,"15min":0.71299999999999997},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.806},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.72199999999999998},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.0940000000000003,"5min":6.734,"15min":10.313000000000001},"last":0.36099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739219,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13435508,"kb_used_data":11561196,"kb_used_omap":1251,"kb_used_meta":1873052,"kb_avail":80936332,"statfs":{"total":96636764160,"available":82878803968,"internally_reserved":0,"allocated":11838664704,"data_stored":32937152671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1918005441},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,2,10,21,4,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.58699999999999997,"15min":0.754},"min":{"1min":0.25800000000000001,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":0.72299999999999998,"5min":8.4030000000000005,"15min":11.715},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.72999999999999998,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.183,"15min":0.183},"max":{"1min":4.2469999999999999,"5min":6.319,"15min":12.058},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:49:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.51500000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":1.522,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55800000000000005,"15min":0.53200000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":1.018,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.505}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11836497920,"data_stored":32935866368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11833876480,"data_stored":32935813120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11834163200,"data_stored":32935800832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:50:44.560 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:44.560 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:44.560 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:44.560 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:50:44.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-09-23T03:50:45.225 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.3s0 on osd.0 to deep-scrub 2023-09-23T03:50:45.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-09-23T03:50:46.105 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2023-09-23T03:50:46.146 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-09-23T03:50:46.975 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.1s0 on osd.0 to deep-scrub 2023-09-23T03:50:47.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-09-23T03:50:47.918 INFO:teuthology.orchestra.run.smithi002.stderr:instructing pg 3.2s0 on osd.1 to deep-scrub 2023-09-23T03:50:47.952 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:50:54.312 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.310+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:50:54.313 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.310+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:50:54.313 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.310+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:50:54.336 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.338+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:50:54.337 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.338+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:50:54.337 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.338+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:50:54.337 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:50:54.338+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:51:07.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:51:08.831 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:51:08.831 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:51:08.898 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2295,"stamp":"2023-09-23T03:51:08.236984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20452129300,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1153196,"num_read_kb":4045646,"num_write":1023237,"num_write_kb":3990621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85498,"ondisk_log_size":85498,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39650364,"kb_used_data":34726756,"kb_used_omap":3754,"kb_used_meta":4919829,"kb_avail":243465156,"statfs":{"total":289910292480,"available":249308319744,"internally_reserved":0,"allocated":35560198144,"data_stored":98811627081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":5037905823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,0,2,8,15,18,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8286208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2033,"num_read_kb":8104,"num_write":1994,"num_write_kb":7976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002171"},"pg_stats":[{"pgid":"2.7","version":"154'7880","reported_seq":12826,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.156840+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:51:05.156840+0000","last_peered":"2023-09-23T03:51:05.156840+0000","last_clean":"2023-09-23T03:51:05.156840+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:51:05.156840+0000","last_undegraded":"2023-09-23T03:51:05.156840+0000","last_fullsized":"2023-09-23T03:51:05.156840+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7880,"log_dups_size":0,"ondisk_log_size":7880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'5333","reported_seq":9202,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:01.289141+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:51:01.289141+0000","last_peered":"2023-09-23T03:51:01.289141+0000","last_clean":"2023-09-23T03:51:01.289141+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:51:01.289141+0000","last_undegraded":"2023-09-23T03:51:01.289141+0000","last_fullsized":"2023-09-23T03:51:01.289141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'6855","reported_seq":11753,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.156487+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:51:05.156487+0000","last_peered":"2023-09-23T03:51:05.156487+0000","last_clean":"2023-09-23T03:51:05.156487+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:51:05.156487+0000","last_undegraded":"2023-09-23T03:51:05.156487+0000","last_fullsized":"2023-09-23T03:51:05.156487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'9493","reported_seq":17206,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.156642+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:51:05.156642+0000","last_peered":"2023-09-23T03:51:05.156642+0000","last_clean":"2023-09-23T03:51:05.156642+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:51:05.156642+0000","last_undegraded":"2023-09-23T03:51:05.156642+0000","last_fullsized":"2023-09-23T03:51:05.156642+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"154'249262","reported_seq":495810,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:04.265781+0000","last_change":"2023-09-23T03:46:09.962567+0000","last_active":"2023-09-23T03:51:04.265781+0000","last_peered":"2023-09-23T03:51:04.265781+0000","last_clean":"2023-09-23T03:51:04.265781+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:51:04.265781+0000","last_undegraded":"2023-09-23T03:51:04.265781+0000","last_fullsized":"2023-09-23T03:51:04.265781+0000","mapping_epoch":18,"log_start":"151'239182","ondisk_log_start":"151'239182","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3855,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 294s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5096968192,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":249058,"num_read_kb":975276,"num_write":246238,"num_write_kb":984952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2254,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-09-23T03:49:03.044271+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:49:03.044271+0000","last_peered":"2023-09-23T03:49:03.044271+0000","last_clean":"2023-09-23T03:49:03.044271+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:49:03.044271+0000","last_undegraded":"2023-09-23T03:49:03.044271+0000","last_fullsized":"2023-09-23T03:49:03.044271+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"154'260758","reported_seq":518729,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:05.701858+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:51:05.701858+0000","last_peered":"2023-09-23T03:51:05.701858+0000","last_clean":"2023-09-23T03:51:05.701858+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:51:05.701858+0000","last_undegraded":"2023-09-23T03:51:05.701858+0000","last_fullsized":"2023-09-23T03:51:05.701858+0000","mapping_epoch":18,"log_start":"152'250698","ondisk_log_start":"152'250698","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2288,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 278s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5330427904,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":260695,"num_read_kb":1020532,"num_write":257577,"num_write_kb":1030308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5716,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.164187+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:51:05.164187+0000","last_peered":"2023-09-23T03:51:05.164187+0000","last_clean":"2023-09-23T03:51:05.164187+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:51:05.164187+0000","last_undegraded":"2023-09-23T03:51:05.164187+0000","last_fullsized":"2023-09-23T03:51:05.164187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"154'254609","reported_seq":506323,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:04.263082+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:51:04.263082+0000","last_peered":"2023-09-23T03:51:04.263082+0000","last_clean":"2023-09-23T03:51:04.263082+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:51:04.263082+0000","last_undegraded":"2023-09-23T03:51:04.263082+0000","last_fullsized":"2023-09-23T03:51:04.263082+0000","mapping_epoch":18,"log_start":"151'244553","ondisk_log_start":"151'244553","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2804,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 288s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5196275712,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":254320,"num_read_kb":996600,"num_write":251525,"num_write_kb":1006100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4900,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.156907+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:51:05.156907+0000","last_peered":"2023-09-23T03:51:05.156907+0000","last_clean":"2023-09-23T03:51:05.156907+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:51:05.156907+0000","last_undegraded":"2023-09-23T03:51:05.156907+0000","last_fullsized":"2023-09-23T03:51:05.156907+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"154'236502","reported_seq":469883,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.678238+0000","last_change":"2023-09-23T03:17:11.829985+0000","last_active":"2023-09-23T03:51:05.678238+0000","last_peered":"2023-09-23T03:51:05.678238+0000","last_clean":"2023-09-23T03:51:05.678238+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:51:05.678238+0000","last_undegraded":"2023-09-23T03:51:05.678238+0000","last_fullsized":"2023-09-23T03:51:05.678238+0000","mapping_epoch":18,"log_start":"151'226472","ondisk_log_start":"151'226472","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4827852819,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":236008,"num_read_kb":924888,"num_write":233591,"num_write_kb":934358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"154'10044","reported_seq":18116,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:05.162242+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:51:05.162242+0000","last_peered":"2023-09-23T03:51:05.162242+0000","last_clean":"2023-09-23T03:51:05.162242+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:51:05.162242+0000","last_undegraded":"2023-09-23T03:51:05.162242+0000","last_fullsized":"2023-09-23T03:51:05.162242+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":324,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:03.888844+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:50:03.888844+0000","last_peered":"2023-09-23T03:50:03.888844+0000","last_clean":"2023-09-23T03:50:03.888844+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:50:03.888844+0000","last_undegraded":"2023-09-23T03:50:03.888844+0000","last_fullsized":"2023-09-23T03:50:03.888844+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20451524627,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1000081,"num_read_kb":3917296,"num_write":988931,"num_write_kb":3955718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35553218560,"data_stored":98808332288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40226,"ondisk_log_size":40226,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45173,"ondisk_log_size":45173,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12947748,"kb_used_data":11575580,"kb_used_omap":1245,"kb_used_meta":1370914,"kb_avail":81424092,"statfs":{"total":96636764160,"available":83378270208,"internally_reserved":0,"allocated":11853393920,"data_stored":32936800316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1403816476},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":0.88800000000000001,"15min":1.0569999999999999},"min":{"1min":0.184,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":14.617000000000001,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":13.618,"5min":13.618,"15min":13.618},"last":0.224}]},{"osd":1,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.48599999999999999,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":10.528,"5min":10.528,"15min":10.528},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.577,"5min":0.78200000000000003,"15min":0.84899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":10.625,"5min":10.625,"15min":10.625},"last":0.85199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739225,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13181400,"kb_used_data":11576464,"kb_used_omap":1256,"kb_used_meta":1603671,"kb_avail":81190440,"statfs":{"total":96636764160,"available":83139010560,"internally_reserved":0,"allocated":11854299136,"data_stored":32937395566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1642159298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,4,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.27400000000000002,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":2.895,"5min":6.093,"15min":10.444000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":6.0529999999999999,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.38600000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:50:06 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.85399999999999998,"15min":0.71299999999999997},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.806},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.82399999999999995,"15min":0.72199999999999998},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":6.0940000000000003,"5min":6.734,"15min":10.313000000000001},"last":0.44600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739223,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13521216,"kb_used_data":11574712,"kb_used_omap":1251,"kb_used_meta":1945244,"kb_avail":80850624,"statfs":{"total":96636764160,"available":82791038976,"internally_reserved":0,"allocated":11852505088,"data_stored":32937431199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1991930049},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,10,15,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.68500000000000005,"15min":0.749},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":6.6740000000000004,"5min":8.4030000000000005,"15min":11.715},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":0.879,"15min":0.91700000000000004},"min":{"1min":0.23499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.782,"5min":6.782,"15min":12.058},"last":0.49099999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.28599999999999998,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":2.819,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11850330112,"data_stored":32936112128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11851313152,"data_stored":32936087552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11851575296,"data_stored":32936132608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:51:08.899 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:08.899 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:08.900 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:08.900 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:08.900 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:51:28.901 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:51:29.663 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:51:29.664 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:51:29.716 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2306,"stamp":"2023-09-23T03:51:28.240643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20469066260,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1157089,"num_read_kb":4061194,"num_write":1027253,"num_write_kb":4006685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85643,"ondisk_log_size":85643,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39908476,"kb_used_data":34768288,"kb_used_omap":3754,"kb_used_meta":5136405,"kb_avail":243207044,"statfs":{"total":289910292480,"available":249044013056,"internally_reserved":0,"allocated":35602726912,"data_stored":98812744376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":5259679647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,9,5,6,17,14,9,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8433664,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1910,"num_read_kb":7624,"num_write":2029,"num_write_kb":8116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002266"},"pg_stats":[{"pgid":"2.7","version":"155'7898","reported_seq":12857,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.647658+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:51:25.647658+0000","last_peered":"2023-09-23T03:51:25.647658+0000","last_clean":"2023-09-23T03:51:25.647658+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:51:25.647658+0000","last_undegraded":"2023-09-23T03:51:25.647658+0000","last_fullsized":"2023-09-23T03:51:25.647658+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'5339","reported_seq":9213,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.632465+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:51:25.632465+0000","last_peered":"2023-09-23T03:51:25.632465+0000","last_clean":"2023-09-23T03:51:25.632465+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:51:25.632465+0000","last_undegraded":"2023-09-23T03:51:25.632465+0000","last_fullsized":"2023-09-23T03:51:25.632465+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'6873","reported_seq":11785,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.643716+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:51:25.643716+0000","last_peered":"2023-09-23T03:51:25.643716+0000","last_clean":"2023-09-23T03:51:25.643716+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:51:25.643716+0000","last_undegraded":"2023-09-23T03:51:25.643716+0000","last_fullsized":"2023-09-23T03:51:25.643716+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'9517","reported_seq":17248,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.665769+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:51:25.665769+0000","last_peered":"2023-09-23T03:51:25.665769+0000","last_clean":"2023-09-23T03:51:25.665769+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:51:25.665769+0000","last_undegraded":"2023-09-23T03:51:25.665769+0000","last_fullsized":"2023-09-23T03:51:25.665769+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9517,"log_dups_size":0,"ondisk_log_size":9517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'250235","reported_seq":497827,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:24.434635+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:51:24.434635+0000","last_peered":"2023-09-23T03:51:24.434635+0000","last_clean":"2023-09-23T03:51:24.434635+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:51:24.434635+0000","last_undegraded":"2023-09-23T03:51:24.434635+0000","last_fullsized":"2023-09-23T03:51:24.434635+0000","mapping_epoch":18,"log_start":"152'240196","ondisk_log_start":"152'240196","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5101043712,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":250085,"num_read_kb":979376,"num_write":247211,"num_write_kb":988844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2256,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:09.342495+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:09.342495+0000","last_peered":"2023-09-23T03:51:09.342495+0000","last_clean":"2023-09-23T03:51:09.342495+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:09.342495+0000","last_undegraded":"2023-09-23T03:51:09.342495+0000","last_fullsized":"2023-09-23T03:51:09.342495+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'261825","reported_seq":520718,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:25.891978+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:51:25.891978+0000","last_peered":"2023-09-23T03:51:25.891978+0000","last_clean":"2023-09-23T03:51:25.891978+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:51:25.891978+0000","last_undegraded":"2023-09-23T03:51:25.891978+0000","last_fullsized":"2023-09-23T03:51:25.891978+0000","mapping_epoch":18,"log_start":"152'251723","ondisk_log_start":"152'251723","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2447,"log_size":10102,"log_dups_size":0,"ondisk_log_size":10102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 298s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5334982656,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":261611,"num_read_kb":1024184,"num_write":258644,"num_write_kb":1034576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5725,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.636228+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:51:25.636228+0000","last_peered":"2023-09-23T03:51:25.636228+0000","last_clean":"2023-09-23T03:51:25.636228+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:51:25.636228+0000","last_undegraded":"2023-09-23T03:51:25.636228+0000","last_fullsized":"2023-09-23T03:51:25.636228+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'255608","reported_seq":508344,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:24.440358+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:51:24.440358+0000","last_peered":"2023-09-23T03:51:24.440358+0000","last_clean":"2023-09-23T03:51:24.440358+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:51:24.440358+0000","last_undegraded":"2023-09-23T03:51:24.440358+0000","last_fullsized":"2023-09-23T03:51:24.440358+0000","mapping_epoch":18,"log_start":"152'245573","ondisk_log_start":"152'245573","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2986,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 308s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5200465920,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":255328,"num_read_kb":1000628,"num_write":252524,"num_write_kb":1010096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4909,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.648229+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:51:25.648229+0000","last_peered":"2023-09-23T03:51:25.648229+0000","last_clean":"2023-09-23T03:51:25.648229+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:51:25.648229+0000","last_undegraded":"2023-09-23T03:51:25.648229+0000","last_fullsized":"2023-09-23T03:51:25.648229+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'237479","reported_seq":471807,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:25.924384+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:51:25.924384+0000","last_peered":"2023-09-23T03:51:25.924384+0000","last_clean":"2023-09-23T03:51:25.924384+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:51:25.924384+0000","last_undegraded":"2023-09-23T03:51:25.924384+0000","last_fullsized":"2023-09-23T03:51:25.924384+0000","mapping_epoch":18,"log_start":"152'227380","ondisk_log_start":"152'227380","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":36,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4831969299,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":236950,"num_read_kb":928656,"num_write":234568,"num_write_kb":938266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'10074","reported_seq":18168,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:25.755137+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:51:25.755137+0000","last_peered":"2023-09-23T03:51:25.755137+0000","last_clean":"2023-09-23T03:51:25.755137+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:51:25.755137+0000","last_undegraded":"2023-09-23T03:51:25.755137+0000","last_fullsized":"2023-09-23T03:51:25.755137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":325,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:25.177734+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:50:25.177734+0000","last_peered":"2023-09-23T03:50:25.177734+0000","last_clean":"2023-09-23T03:50:25.177734+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:50:25.177734+0000","last_undegraded":"2023-09-23T03:50:25.177734+0000","last_fullsized":"2023-09-23T03:50:25.177734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20468461587,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1003974,"num_read_kb":3932844,"num_write":992947,"num_write_kb":3971782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35596443648,"data_stored":98809569280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40275,"ondisk_log_size":40275,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45269,"ondisk_log_size":45269,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739231,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13033964,"kb_used_data":11589600,"kb_used_omap":1245,"kb_used_meta":1443106,"kb_avail":81337876,"statfs":{"total":96636764160,"available":83289985024,"internally_reserved":0,"allocated":11867750400,"data_stored":32937198689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1477741084},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":0.88800000000000001,"15min":1.0569999999999999},"min":{"1min":0.184,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":14.617000000000001,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":13.618,"5min":13.618,"15min":13.618},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.48599999999999999,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":10.528,"5min":10.528,"15min":10.528},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.577,"5min":0.78200000000000003,"15min":0.84899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":10.625,"5min":10.625,"15min":10.625},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739229,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13285124,"kb_used_data":11589948,"kb_used_omap":1256,"kb_used_meta":1693911,"kb_avail":81086716,"statfs":{"total":96636764160,"available":83032797184,"internally_reserved":0,"allocated":11868106752,"data_stored":32937798035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1734565058},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,8,6,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":1.581,"5min":6.093,"15min":7.0410000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.64300000000000002,"15min":0.71399999999999997},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":1.661,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.87}]},{"osd":2,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":3.585,"5min":6.0810000000000004,"15min":6.806},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.749,"15min":0.71099999999999997},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":3.4580000000000002,"5min":6.734,"15min":7.9210000000000003},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13589388,"kb_used_data":11588740,"kb_used_omap":1251,"kb_used_meta":1999388,"kb_avail":80782452,"statfs":{"total":96636764160,"available":82721230848,"internally_reserved":0,"allocated":11866869760,"data_stored":32937747652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":2047373505},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,5,4,1,4,8,3,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.68500000000000005,"15min":0.749},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":6.6740000000000004,"5min":8.4030000000000005,"15min":11.715},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":0.879,"15min":0.91700000000000004},"min":{"1min":0.23499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.782,"5min":6.782,"15min":12.058},"last":0.245}]},{"osd":2,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.28599999999999998,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":2.819,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11864768512,"data_stored":32936521728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11865755648,"data_stored":32936562688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11865919488,"data_stored":32936484864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:51:29.718 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:29.718 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:29.718 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:29.719 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:51:36.652 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.649+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:51:36.652 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.649+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:51:36.652 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.649+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:51:36.653 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.649+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:51:36.653 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.653+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:51:36.653 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.653+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:51:36.653 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:51:36.653+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:51:49.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:51:50.391 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:51:50.391 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:51:50.438 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2317,"stamp":"2023-09-23T03:51:50.248558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20487170580,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1161281,"num_read_kb":4077922,"num_write":1031591,"num_write_kb":4024037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85659,"ondisk_log_size":85659,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39447464,"kb_used_data":34812616,"kb_used_omap":3754,"kb_used_meta":4631061,"kb_avail":243668056,"statfs":{"total":289910292480,"available":249516089344,"internally_reserved":0,"allocated":35648118784,"data_stored":98813555384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4742207391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,11,42,19,18,16,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9801728,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2254,"num_read_kb":9004,"num_write":2330,"num_write_kb":9320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006173"},"pg_stats":[{"pgid":"2.7","version":"155'7907","reported_seq":12878,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.119186+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:51:46.119186+0000","last_peered":"2023-09-23T03:51:46.119186+0000","last_clean":"2023-09-23T03:51:46.119186+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:51:46.119186+0000","last_undegraded":"2023-09-23T03:51:46.119186+0000","last_fullsized":"2023-09-23T03:51:46.119186+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7907,"log_dups_size":0,"ondisk_log_size":7907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'5342","reported_seq":9220,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:44.959767+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:51:44.959767+0000","last_peered":"2023-09-23T03:51:44.959767+0000","last_clean":"2023-09-23T03:51:44.959767+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:51:44.959767+0000","last_undegraded":"2023-09-23T03:51:44.959767+0000","last_fullsized":"2023-09-23T03:51:44.959767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'6882","reported_seq":11806,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.119221+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:51:46.119221+0000","last_peered":"2023-09-23T03:51:46.119221+0000","last_clean":"2023-09-23T03:51:46.119221+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:51:46.119221+0000","last_undegraded":"2023-09-23T03:51:46.119221+0000","last_fullsized":"2023-09-23T03:51:46.119221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6882,"log_dups_size":0,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'9529","reported_seq":17276,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.119555+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:51:46.119555+0000","last_peered":"2023-09-23T03:51:46.119555+0000","last_clean":"2023-09-23T03:51:46.119555+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:51:46.119555+0000","last_undegraded":"2023-09-23T03:51:46.119555+0000","last_fullsized":"2023-09-23T03:51:46.119555+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'251421","reported_seq":500215,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:49.822045+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:51:49.822045+0000","last_peered":"2023-09-23T03:51:49.822045+0000","last_clean":"2023-09-23T03:51:49.822045+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:51:49.822045+0000","last_undegraded":"2023-09-23T03:51:49.822045+0000","last_fullsized":"2023-09-23T03:51:49.822045+0000","mapping_epoch":18,"log_start":"152'241408","ondisk_log_start":"152'241408","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5105917952,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":251284,"num_read_kb":984156,"num_write":248397,"num_write_kb":993588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2256,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:09.342495+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:09.342495+0000","last_peered":"2023-09-23T03:51:09.342495+0000","last_clean":"2023-09-23T03:51:09.342495+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:09.342495+0000","last_undegraded":"2023-09-23T03:51:09.342495+0000","last_fullsized":"2023-09-23T03:51:09.342495+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'262839","reported_seq":522666,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:46.127876+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:51:46.127876+0000","last_peered":"2023-09-23T03:51:46.127876+0000","last_clean":"2023-09-23T03:51:46.127876+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:51:46.127876+0000","last_undegraded":"2023-09-23T03:51:46.127876+0000","last_fullsized":"2023-09-23T03:51:46.127876+0000","mapping_epoch":18,"log_start":"152'252744","ondisk_log_start":"152'252744","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2601,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 318s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5339242496,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":262532,"num_read_kb":1027852,"num_write":259658,"num_write_kb":1038632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5733,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.122038+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:51:46.122038+0000","last_peered":"2023-09-23T03:51:46.122038+0000","last_clean":"2023-09-23T03:51:46.122038+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:51:46.122038+0000","last_undegraded":"2023-09-23T03:51:46.122038+0000","last_fullsized":"2023-09-23T03:51:46.122038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'256839","reported_seq":510821,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:49.820365+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:51:49.820365+0000","last_peered":"2023-09-23T03:51:49.820365+0000","last_clean":"2023-09-23T03:51:49.820365+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:51:49.820365+0000","last_undegraded":"2023-09-23T03:51:49.820365+0000","last_fullsized":"2023-09-23T03:51:49.820365+0000","mapping_epoch":18,"log_start":"152'246796","ondisk_log_start":"152'246796","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3192,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5205622784,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":256561,"num_read_kb":1005552,"num_write":253755,"num_write_kb":1015020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4917,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.141917+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:51:46.141917+0000","last_peered":"2023-09-23T03:51:46.141917+0000","last_clean":"2023-09-23T03:51:46.141917+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:51:46.141917+0000","last_undegraded":"2023-09-23T03:51:46.141917+0000","last_fullsized":"2023-09-23T03:51:46.141917+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'238386","reported_seq":473562,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:51:46.027713+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:51:46.027713+0000","last_peered":"2023-09-23T03:51:46.027713+0000","last_clean":"2023-09-23T03:51:46.027713+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:51:46.027713+0000","last_undegraded":"2023-09-23T03:51:46.027713+0000","last_fullsized":"2023-09-23T03:51:46.027713+0000","mapping_epoch":18,"log_start":"152'228294","ondisk_log_start":"152'228294","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":295,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 22s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4835782675,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":237789,"num_read_kb":932012,"num_write":235475,"num_write_kb":941894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'10089","reported_seq":18203,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:46.120785+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:51:46.120785+0000","last_peered":"2023-09-23T03:51:46.120785+0000","last_clean":"2023-09-23T03:51:46.120785+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:51:46.120785+0000","last_undegraded":"2023-09-23T03:51:46.120785+0000","last_fullsized":"2023-09-23T03:51:46.120785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":325,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:25.177734+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:50:25.177734+0000","last_peered":"2023-09-23T03:50:25.177734+0000","last_clean":"2023-09-23T03:50:25.177734+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:50:25.177734+0000","last_undegraded":"2023-09-23T03:50:25.177734+0000","last_fullsized":"2023-09-23T03:50:25.177734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20486565907,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1008166,"num_read_kb":3949572,"num_write":997285,"num_write_kb":3989134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35640836096,"data_stored":98810359808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40243,"ondisk_log_size":40243,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45317,"ondisk_log_size":45317,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13101224,"kb_used_data":11602720,"kb_used_omap":1245,"kb_used_meta":1497250,"kb_avail":81270616,"statfs":{"total":96636764160,"available":83221110784,"internally_reserved":0,"allocated":11881185280,"data_stored":32937444449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1533184540},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":0.88800000000000001,"15min":1.0569999999999999},"min":{"1min":0.184,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":14.617000000000001,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.57599999999999996,"15min":0.60299999999999998},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":13.618,"5min":13.618,"15min":13.618},"last":0.29999999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:51:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.48599999999999999,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23300000000000001,"15min":0.191},"max":{"1min":10.528,"5min":10.528,"15min":10.528},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.577,"5min":0.78200000000000003,"15min":0.84899999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":10.625,"5min":10.625,"15min":10.625},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739233,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13357380,"kb_used_data":11603956,"kb_used_omap":1256,"kb_used_meta":1752151,"kb_avail":81014460,"statfs":{"total":96636764160,"available":82958807040,"internally_reserved":0,"allocated":11882450944,"data_stored":32938051987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1794202818},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,39,13,12,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":1.581,"5min":6.093,"15min":7.0410000000000004},"last":2.9020000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.64300000000000002,"15min":0.71399999999999997},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":1.661,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":2.827}]},{"osd":2,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":3.585,"5min":6.0810000000000004,"15min":6.806},"last":2.863},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.749,"15min":0.71099999999999997},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":3.4580000000000002,"5min":6.734,"15min":7.9210000000000003},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12988860,"kb_used_data":11605940,"kb_used_omap":1251,"kb_used_meta":1381660,"kb_avail":81382980,"statfs":{"total":96636764160,"available":83336171520,"internally_reserved":0,"allocated":11884482560,"data_stored":32938058948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1414820033},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,3,0,4,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.68500000000000005,"15min":0.749},"min":{"1min":0.253,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":6.6740000000000004,"5min":8.4030000000000005,"15min":11.715},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.3129999999999999,"5min":0.879,"15min":0.91700000000000004},"min":{"1min":0.23499999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.782,"5min":6.782,"15min":12.058},"last":7.0469999999999997}]},{"osd":2,"last update":"Sat Sep 23 03:50:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.28599999999999998,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":2.819,"5min":5.6820000000000004,"15min":5.6820000000000004},"last":8.5700000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.55100000000000005,"15min":0.53300000000000003},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57599999999999996,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11881807872,"data_stored":32936812544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11879440384,"data_stored":32936767488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11879587840,"data_stored":32936779776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:51:50.440 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:50.440 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:50.440 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:51:50.440 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:52:10.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:52:11.933 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:52:11.934 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:52:11.973 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2327,"stamp":"2023-09-23T03:52:10.256920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20503296532,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1164968,"num_read_kb":4092630,"num_write":1035528,"num_write_kb":4039785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85713,"ondisk_log_size":85713,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38941476,"kb_used_data":34852236,"kb_used_omap":3754,"kb_used_meta":4085461,"kb_avail":244174044,"statfs":{"total":289910292480,"available":250034221056,"internally_reserved":0,"allocated":35688689664,"data_stored":98814120632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4183512991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,2,1,11,15,11,18,11,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10358784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2362,"num_read_kb":9424,"num_write":2531,"num_write_kb":10124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006938"},"pg_stats":[{"pgid":"2.7","version":"155'7940","reported_seq":12921,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:05.234041+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:52:05.234041+0000","last_peered":"2023-09-23T03:52:05.234041+0000","last_clean":"2023-09-23T03:52:05.234041+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:52:05.234041+0000","last_undegraded":"2023-09-23T03:52:05.234041+0000","last_fullsized":"2023-09-23T03:52:05.234041+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7940,"log_dups_size":0,"ondisk_log_size":7940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'5353","reported_seq":9234,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:04.849319+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:52:04.849319+0000","last_peered":"2023-09-23T03:52:04.849319+0000","last_clean":"2023-09-23T03:52:04.849319+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:52:04.849319+0000","last_undegraded":"2023-09-23T03:52:04.849319+0000","last_fullsized":"2023-09-23T03:52:04.849319+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'6915","reported_seq":11848,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:05.240108+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:52:05.240108+0000","last_peered":"2023-09-23T03:52:05.240108+0000","last_clean":"2023-09-23T03:52:05.240108+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:52:05.240108+0000","last_undegraded":"2023-09-23T03:52:05.240108+0000","last_fullsized":"2023-09-23T03:52:05.240108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6915,"log_dups_size":0,"ondisk_log_size":6915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'9573","reported_seq":17332,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:05.248146+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:52:05.248146+0000","last_peered":"2023-09-23T03:52:05.248146+0000","last_clean":"2023-09-23T03:52:05.248146+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:52:05.248146+0000","last_undegraded":"2023-09-23T03:52:05.248146+0000","last_fullsized":"2023-09-23T03:52:05.248146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"155'252455","reported_seq":502156,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:10.101729+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:52:10.101729+0000","last_peered":"2023-09-23T03:52:10.101729+0000","last_clean":"2023-09-23T03:52:10.101729+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:52:10.101729+0000","last_undegraded":"2023-09-23T03:52:10.101729+0000","last_fullsized":"2023-09-23T03:52:10.101729+0000","mapping_epoch":18,"log_start":"152'242421","ondisk_log_start":"152'242421","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5110202368,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":252195,"num_read_kb":987788,"num_write":249431,"num_write_kb":997724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2256,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:51:09.342495+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:09.342495+0000","last_peered":"2023-09-23T03:51:09.342495+0000","last_clean":"2023-09-23T03:51:09.342495+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:09.342495+0000","last_undegraded":"2023-09-23T03:51:09.342495+0000","last_fullsized":"2023-09-23T03:51:09.342495+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"155'263864","reported_seq":524677,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:06.376580+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:52:06.376580+0000","last_peered":"2023-09-23T03:52:06.376580+0000","last_clean":"2023-09-23T03:52:06.376580+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:52:06.376580+0000","last_undegraded":"2023-09-23T03:52:06.376580+0000","last_fullsized":"2023-09-23T03:52:06.376580+0000","mapping_epoch":18,"log_start":"152'253765","ondisk_log_start":"152'253765","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2719,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 339s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5343416320,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":263504,"num_read_kb":1031736,"num_write":260683,"num_write_kb":1042732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"150'2135","reported_seq":5739,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:01.476520+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:52:01.476520+0000","last_peered":"2023-09-23T03:52:01.476520+0000","last_clean":"2023-09-23T03:52:01.476520+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:52:01.476520+0000","last_undegraded":"2023-09-23T03:52:01.476520+0000","last_fullsized":"2023-09-23T03:52:01.476520+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2135,"log_dups_size":0,"ondisk_log_size":2135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'257835","reported_seq":512797,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:10.100223+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:52:10.100223+0000","last_peered":"2023-09-23T03:52:10.100223+0000","last_clean":"2023-09-23T03:52:10.100223+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:52:10.100223+0000","last_undegraded":"2023-09-23T03:52:10.100223+0000","last_fullsized":"2023-09-23T03:52:10.100223+0000","mapping_epoch":18,"log_start":"152'247808","ondisk_log_start":"152'247808","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3418,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 354s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5209702400,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":257532,"num_read_kb":1009428,"num_write":254751,"num_write_kb":1019004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"150'1505","reported_seq":4923,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:01.491052+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:52:01.491052+0000","last_peered":"2023-09-23T03:52:01.491052+0000","last_clean":"2023-09-23T03:52:01.491052+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:52:01.491052+0000","last_undegraded":"2023-09-23T03:52:01.491052+0000","last_fullsized":"2023-09-23T03:52:01.491052+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"155'239268","reported_seq":475304,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:06.385224+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:52:06.385224+0000","last_peered":"2023-09-23T03:52:06.385224+0000","last_clean":"2023-09-23T03:52:06.385224+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:52:06.385224+0000","last_undegraded":"2023-09-23T03:52:06.385224+0000","last_fullsized":"2023-09-23T03:52:06.385224+0000","mapping_epoch":18,"log_start":"152'229207","ondisk_log_start":"152'229207","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":602,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 43s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4839370771,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":238622,"num_read_kb":935328,"num_write":236357,"num_write_kb":945422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"155'10144","reported_seq":18273,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:05.320836+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:52:05.320836+0000","last_peered":"2023-09-23T03:52:05.320836+0000","last_clean":"2023-09-23T03:52:05.320836+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:52:05.320836+0000","last_undegraded":"2023-09-23T03:52:05.320836+0000","last_fullsized":"2023-09-23T03:52:05.320836+0000","mapping_epoch":13,"log_start":"43'100","ondisk_log_start":"43'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":325,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-09-23T03:50:25.177734+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:50:25.177734+0000","last_peered":"2023-09-23T03:50:25.177734+0000","last_clean":"2023-09-23T03:50:25.177734+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:50:25.177734+0000","last_undegraded":"2023-09-23T03:50:25.177734+0000","last_fullsized":"2023-09-23T03:50:25.177734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20502691859,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1011853,"num_read_kb":3964280,"num_write":1001222,"num_write_kb":4004882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35681820672,"data_stored":98810822656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40221,"ondisk_log_size":40221,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45393,"ondisk_log_size":45393,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739239,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13168400,"kb_used_data":11615752,"kb_used_omap":1245,"kb_used_meta":1551394,"kb_avail":81203440,"statfs":{"total":96636764160,"available":83152322560,"internally_reserved":0,"allocated":11894530048,"data_stored":32937645153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1588627996},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":0.93999999999999995,"15min":1.0509999999999999},"min":{"1min":0.192,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":12.862,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.192},{"interface":"front","average":{"1min":0.309,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.5,"5min":13.618,"15min":13.618},"last":0.20799999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.52400000000000002,"15min":0.60799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.20000000000000001},"max":{"1min":4.5780000000000003,"5min":10.528,"15min":10.528},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":4.5359999999999996,"5min":10.625,"15min":10.625},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13442776,"kb_used_data":11617168,"kb_used_omap":1256,"kb_used_meta":1824343,"kb_avail":80929064,"statfs":{"total":96636764160,"available":82871361536,"internally_reserved":0,"allocated":11895980032,"data_stored":32938240403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1868127426},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,6,7,14,10,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.86699999999999999,"15min":0.872},"min":{"1min":0.20300000000000001,"5min":0.19700000000000001,"15min":0.161},"max":{"1min":1.581,"5min":6.093,"15min":7.0410000000000004},"last":0.371},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.64300000000000002,"15min":0.71399999999999997},"min":{"1min":0.217,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":1.661,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:51:07 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.21099999999999999},"max":{"1min":3.585,"5min":6.0810000000000004,"15min":6.806},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.749,"15min":0.71099999999999997},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":3.4580000000000002,"5min":6.734,"15min":7.9210000000000003},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359739236,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12330300,"kb_used_data":11619316,"kb_used_omap":1251,"kb_used_meta":709724,"kb_avail":82041540,"statfs":{"total":96636764160,"available":84010536960,"internally_reserved":0,"allocated":11898179584,"data_stored":32938235076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":726757569},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,8,6,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.80800000000000005,"15min":0.78900000000000003},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":7.0880000000000001,"5min":8.4030000000000005,"15min":11.715},"last":0.38},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.0189999999999999,"15min":0.97999999999999998},"min":{"1min":0.215,"5min":0.183,"15min":0.183},"max":{"1min":7.0469999999999997,"5min":7.0469999999999997,"15min":12.058},"last":0.22500000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":0.68999999999999995,"15min":0.56699999999999995},"min":{"1min":0.245,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.441},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.59299999999999997,"15min":0.56000000000000005},"min":{"1min":0.33200000000000002,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11895685120,"data_stored":32937005056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11893030912,"data_stored":32936890368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11893104640,"data_stored":32936927232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:52:11.975 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:11.976 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:11.976 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:11.976 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:52:31.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:52:32.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:52:32.736 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:52:32.812 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2339,"stamp":"2023-09-23T03:52:32.260870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20521753108,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1169126,"num_read_kb":4109246,"num_write":1039999,"num_write_kb":4057669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85827,"ondisk_log_size":85827,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39290280,"kb_used_data":34899852,"kb_used_omap":3754,"kb_used_meta":4386645,"kb_avail":243825240,"statfs":{"total":289910292480,"available":249677045760,"internally_reserved":0,"allocated":35737448448,"data_stored":98814877350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4491925407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,5,7,12,24,20,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8376320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1837,"num_read_kb":7340,"num_write":2028,"num_write_kb":8112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001765"},"pg_stats":[{"pgid":"2.7","version":"156'7961","reported_seq":12958,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:30.827683+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:52:30.827683+0000","last_peered":"2023-09-23T03:52:30.827683+0000","last_clean":"2023-09-23T03:52:30.827683+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:52:30.827683+0000","last_undegraded":"2023-09-23T03:52:30.827683+0000","last_fullsized":"2023-09-23T03:52:30.827683+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'5360","reported_seq":9248,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:28.885329+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:52:28.885329+0000","last_peered":"2023-09-23T03:52:28.885329+0000","last_clean":"2023-09-23T03:52:28.885329+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:52:28.885329+0000","last_undegraded":"2023-09-23T03:52:28.885329+0000","last_fullsized":"2023-09-23T03:52:28.885329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'6936","reported_seq":11885,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:30.820590+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:52:30.820590+0000","last_peered":"2023-09-23T03:52:30.820590+0000","last_clean":"2023-09-23T03:52:30.820590+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:52:30.820590+0000","last_undegraded":"2023-09-23T03:52:30.820590+0000","last_fullsized":"2023-09-23T03:52:30.820590+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'9601","reported_seq":17381,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:30.845941+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:52:30.845941+0000","last_peered":"2023-09-23T03:52:30.845941+0000","last_clean":"2023-09-23T03:52:30.845941+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:52:30.845941+0000","last_undegraded":"2023-09-23T03:52:30.845941+0000","last_fullsized":"2023-09-23T03:52:30.845941+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'253487","reported_seq":504152,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:30.408643+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:52:30.408643+0000","last_peered":"2023-09-23T03:52:30.408643+0000","last_clean":"2023-09-23T03:52:30.408643+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:52:30.408643+0000","last_undegraded":"2023-09-23T03:52:30.408643+0000","last_fullsized":"2023-09-23T03:52:30.408643+0000","mapping_epoch":18,"log_start":"153'243442","ondisk_log_start":"153'243442","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5114429440,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":253155,"num_read_kb":991620,"num_write":250463,"num_write_kb":1001852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2257,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:13.865746+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:13.865746+0000","last_peered":"2023-09-23T03:51:13.865746+0000","last_clean":"2023-09-23T03:51:13.865746+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:13.865746+0000","last_undegraded":"2023-09-23T03:51:13.865746+0000","last_fullsized":"2023-09-23T03:51:13.865746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'265160","reported_seq":527163,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:31.686756+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:52:31.686756+0000","last_peered":"2023-09-23T03:52:31.686756+0000","last_clean":"2023-09-23T03:52:31.686756+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:52:31.686756+0000","last_undegraded":"2023-09-23T03:52:31.686756+0000","last_fullsized":"2023-09-23T03:52:31.686756+0000","mapping_epoch":18,"log_start":"153'255085","ondisk_log_start":"153'255085","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2999,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 364s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5348704256,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":264680,"num_read_kb":1036436,"num_write":261979,"num_write_kb":1047916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5750,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:27.109743+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:52:27.109743+0000","last_peered":"2023-09-23T03:52:27.109743+0000","last_clean":"2023-09-23T03:52:27.109743+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:52:27.109743+0000","last_undegraded":"2023-09-23T03:52:27.109743+0000","last_fullsized":"2023-09-23T03:52:27.109743+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'258855","reported_seq":514800,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:30.360487+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:52:30.360487+0000","last_peered":"2023-09-23T03:52:30.360487+0000","last_clean":"2023-09-23T03:52:30.360487+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:52:30.360487+0000","last_undegraded":"2023-09-23T03:52:30.360487+0000","last_fullsized":"2023-09-23T03:52:30.360487+0000","mapping_epoch":18,"log_start":"153'248819","ondisk_log_start":"153'248819","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3585,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 374s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5213904896,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":258497,"num_read_kb":1013284,"num_write":255771,"num_write_kb":1023084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4935,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:27.081860+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:52:27.081860+0000","last_peered":"2023-09-23T03:52:27.081860+0000","last_clean":"2023-09-23T03:52:27.081860+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:52:27.081860+0000","last_undegraded":"2023-09-23T03:52:27.081860+0000","last_fullsized":"2023-09-23T03:52:27.081860+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'240391","reported_seq":477500,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:31.689745+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:52:31.689745+0000","last_peered":"2023-09-23T03:52:31.689745+0000","last_clean":"2023-09-23T03:52:31.689745+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:52:31.689745+0000","last_undegraded":"2023-09-23T03:52:31.689745+0000","last_fullsized":"2023-09-23T03:52:31.689745+0000","mapping_epoch":18,"log_start":"152'230328","ondisk_log_start":"152'230328","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":970,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 68s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4844109843,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":239679,"num_read_kb":939556,"num_write":237480,"num_write_kb":949914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'10179","reported_seq":18334,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:30.883337+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:52:30.883337+0000","last_peered":"2023-09-23T03:52:30.883337+0000","last_clean":"2023-09-23T03:52:30.883337+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:52:30.883337+0000","last_undegraded":"2023-09-23T03:52:30.883337+0000","last_fullsized":"2023-09-23T03:52:30.883337+0000","mapping_epoch":13,"log_start":"43'100","ondisk_log_start":"43'100","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":327,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:26.631129+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:26.631129+0000","last_peered":"2023-09-23T03:52:26.631129+0000","last_clean":"2023-09-23T03:52:26.631129+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:26.631129+0000","last_undegraded":"2023-09-23T03:52:26.631129+0000","last_fullsized":"2023-09-23T03:52:26.631129+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20521148435,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1016011,"num_read_kb":3980896,"num_write":1005693,"num_write_kb":4022766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35730923520,"data_stored":98811502592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40219,"ondisk_log_size":40219,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45509,"ondisk_log_size":45509,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739244,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13279764,"kb_used_data":11632716,"kb_used_omap":1245,"kb_used_meta":1645794,"kb_avail":81092076,"statfs":{"total":96636764160,"available":83038285824,"internally_reserved":0,"allocated":11911901184,"data_stored":32937887835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1685293596},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,2,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":0.93999999999999995,"15min":1.0509999999999999},"min":{"1min":0.192,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":12.862,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.754},{"interface":"front","average":{"1min":0.309,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.5,"5min":13.618,"15min":13.618},"last":0.314}]},{"osd":1,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.52400000000000002,"15min":0.60799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.20000000000000001},"max":{"1min":4.5780000000000003,"5min":10.528,"15min":10.528},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":4.5359999999999996,"5min":10.625,"15min":10.625},"last":0.38}]}]},{"osd":1,"up_from":8,"seq":34359739242,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13590284,"kb_used_data":11634244,"kb_used_omap":1256,"kb_used_meta":1954775,"kb_avail":80781556,"statfs":{"total":96636764160,"available":82720313344,"internally_reserved":0,"allocated":11913465856,"data_stored":32938503565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":2001689794},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,7,15,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82399999999999995,"15min":0.89600000000000002},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":7.0410000000000004},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.624,"15min":0.72499999999999998},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":2.827,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.372}]},{"osd":2,"last update":"Sat Sep 23 03:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.863,"5min":6.0810000000000004,"15min":6.806},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.755,"15min":0.70799999999999996},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":0.55200000000000005,"5min":6.734,"15min":7.9210000000000003},"last":0.41099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739240,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12420232,"kb_used_data":11632892,"kb_used_omap":1251,"kb_used_meta":786076,"kb_avail":81951608,"statfs":{"total":96636764160,"available":83918446592,"internally_reserved":0,"allocated":11912081408,"data_stored":32938485950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":804942017},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.80800000000000005,"15min":0.78900000000000003},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":7.0880000000000001,"5min":8.4030000000000005,"15min":11.715},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.0189999999999999,"15min":0.97999999999999998},"min":{"1min":0.215,"5min":0.183,"15min":0.183},"max":{"1min":7.0469999999999997,"5min":7.0469999999999997,"15min":12.058},"last":0.317}]},{"osd":2,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":0.68999999999999995,"15min":0.56699999999999995},"min":{"1min":0.245,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.59299999999999997,"15min":0.56000000000000005},"min":{"1min":0.33200000000000002,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11909718016,"data_stored":32937152512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11910639616,"data_stored":32937185280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11910565888,"data_stored":32937164800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:52:32.814 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:32.814 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:32.814 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:32.814 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:52:52.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:52:53.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:52:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:52:53.636 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2349,"stamp":"2023-09-23T03:52:52.264518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20538120724,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1172861,"num_read_kb":4124174,"num_write":1043996,"num_write_kb":4073657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85852,"ondisk_log_size":85852,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39547676,"kb_used_data":34940732,"kb_used_omap":3754,"kb_used_meta":4603157,"kb_avail":243567844,"statfs":{"total":289910292480,"available":249413472256,"internally_reserved":0,"allocated":35779309568,"data_stored":98815078054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4713633695},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,4,4,21,13,11,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12410880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2804,"num_read_kb":11208,"num_write":3021,"num_write_kb":12084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.7","version":"156'7976","reported_seq":12985,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:50.154647+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:52:50.154647+0000","last_peered":"2023-09-23T03:52:50.154647+0000","last_clean":"2023-09-23T03:52:50.154647+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:52:50.154647+0000","last_undegraded":"2023-09-23T03:52:50.154647+0000","last_fullsized":"2023-09-23T03:52:50.154647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":7976,"log_dups_size":0,"ondisk_log_size":7976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'5365","reported_seq":9257,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:47.560725+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:52:47.560725+0000","last_peered":"2023-09-23T03:52:47.560725+0000","last_clean":"2023-09-23T03:52:47.560725+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:52:47.560725+0000","last_undegraded":"2023-09-23T03:52:47.560725+0000","last_fullsized":"2023-09-23T03:52:47.560725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'6951","reported_seq":11912,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:50.143916+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:52:50.143916+0000","last_peered":"2023-09-23T03:52:50.143916+0000","last_clean":"2023-09-23T03:52:50.143916+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:52:50.143916+0000","last_undegraded":"2023-09-23T03:52:50.143916+0000","last_fullsized":"2023-09-23T03:52:50.143916+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'9621","reported_seq":17417,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:50.154776+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:52:50.154776+0000","last_peered":"2023-09-23T03:52:50.154776+0000","last_clean":"2023-09-23T03:52:50.154776+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:52:50.154776+0000","last_undegraded":"2023-09-23T03:52:50.154776+0000","last_fullsized":"2023-09-23T03:52:50.154776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'254518","reported_seq":506103,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:50.524286+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:52:50.524286+0000","last_peered":"2023-09-23T03:52:50.524286+0000","last_clean":"2023-09-23T03:52:50.524286+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:52:50.524286+0000","last_undegraded":"2023-09-23T03:52:50.524286+0000","last_fullsized":"2023-09-23T03:52:50.524286+0000","mapping_epoch":18,"log_start":"153'244453","ondisk_log_start":"153'244453","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5118623744,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":254074,"num_read_kb":995292,"num_write":251494,"num_write_kb":1005976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2257,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:13.865746+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:13.865746+0000","last_peered":"2023-09-23T03:51:13.865746+0000","last_clean":"2023-09-23T03:51:13.865746+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:13.865746+0000","last_undegraded":"2023-09-23T03:51:13.865746+0000","last_fullsized":"2023-09-23T03:51:13.865746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'266151","reported_seq":529133,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:51.937711+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:52:51.937711+0000","last_peered":"2023-09-23T03:52:51.937711+0000","last_clean":"2023-09-23T03:52:51.937711+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:52:51.937711+0000","last_undegraded":"2023-09-23T03:52:51.937711+0000","last_fullsized":"2023-09-23T03:52:51.937711+0000","mapping_epoch":18,"log_start":"153'256101","ondisk_log_start":"153'256101","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3206,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 384s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5352804352,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":265644,"num_read_kb":1040292,"num_write":262970,"num_write_kb":1051880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5759,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:47.562197+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:52:47.562197+0000","last_peered":"2023-09-23T03:52:47.562197+0000","last_clean":"2023-09-23T03:52:47.562197+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:52:47.562197+0000","last_undegraded":"2023-09-23T03:52:47.562197+0000","last_fullsized":"2023-09-23T03:52:47.562197+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'259904","reported_seq":516757,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:50.498909+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:52:50.498909+0000","last_peered":"2023-09-23T03:52:50.498909+0000","last_clean":"2023-09-23T03:52:50.498909+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:52:50.498909+0000","last_undegraded":"2023-09-23T03:52:50.498909+0000","last_fullsized":"2023-09-23T03:52:50.498909+0000","mapping_epoch":18,"log_start":"153'249836","ondisk_log_start":"153'249836","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3715,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 394s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5218160640,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":259399,"num_read_kb":1016884,"num_write":256820,"num_write_kb":1027280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4943,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:47.564070+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:52:47.564070+0000","last_peered":"2023-09-23T03:52:47.564070+0000","last_clean":"2023-09-23T03:52:47.564070+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:52:47.564070+0000","last_undegraded":"2023-09-23T03:52:47.564070+0000","last_fullsized":"2023-09-23T03:52:47.564070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'241317","reported_seq":479397,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:52:51.956580+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:52:51.956580+0000","last_peered":"2023-09-23T03:52:51.956580+0000","last_clean":"2023-09-23T03:52:51.956580+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:52:51.956580+0000","last_undegraded":"2023-09-23T03:52:51.956580+0000","last_fullsized":"2023-09-23T03:52:51.956580+0000","mapping_epoch":18,"log_start":"153'231236","ondisk_log_start":"153'231236","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1214,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 88s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4847927315,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":240629,"num_read_kb":943356,"num_write":238406,"num_write_kb":953618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'10204","reported_seq":18379,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:50.223768+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:52:50.223768+0000","last_peered":"2023-09-23T03:52:50.223768+0000","last_clean":"2023-09-23T03:52:50.223768+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:52:50.223768+0000","last_undegraded":"2023-09-23T03:52:50.223768+0000","last_fullsized":"2023-09-23T03:52:50.223768+0000","mapping_epoch":13,"log_start":"43'200","ondisk_log_start":"43'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":327,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:26.631129+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:26.631129+0000","last_peered":"2023-09-23T03:52:26.631129+0000","last_clean":"2023-09-23T03:52:26.631129+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:26.631129+0000","last_undegraded":"2023-09-23T03:52:26.631129+0000","last_fullsized":"2023-09-23T03:52:26.631129+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20537516051,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1019746,"num_read_kb":3995824,"num_write":1009690,"num_write_kb":4038754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35773009920,"data_stored":98811858944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40264,"ondisk_log_size":40264,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45489,"ondisk_log_size":45489,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739248,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13365496,"kb_used_data":11646320,"kb_used_omap":1245,"kb_used_meta":1717922,"kb_avail":81006344,"statfs":{"total":96636764160,"available":82950496256,"internally_reserved":0,"allocated":11925831680,"data_stored":32937957467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1759152668},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":0.93999999999999995,"15min":1.0509999999999999},"min":{"1min":0.192,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":12.862,"5min":14.617000000000001,"15min":22.789000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.309,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.20799999999999999,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.5,"5min":13.618,"15min":13.618},"last":0.42399999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:52:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.52400000000000002,"15min":0.60799999999999998},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.20000000000000001},"max":{"1min":4.5780000000000003,"5min":10.528,"15min":10.528},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.83399999999999996,"15min":0.84999999999999998},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":4.5359999999999996,"5min":10.625,"15min":10.625},"last":0.502}]}]},{"osd":1,"up_from":8,"seq":34359739246,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13676224,"kb_used_data":11647984,"kb_used_omap":1256,"kb_used_meta":2026967,"kb_avail":80695616,"statfs":{"total":96636764160,"available":82632310784,"internally_reserved":0,"allocated":11927535616,"data_stored":32938581389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":2075614402},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,11,3,6,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.82399999999999995,"15min":0.89600000000000002},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.161},"max":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":7.0410000000000004},"last":2.456},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.624,"15min":0.72499999999999998},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":2.827,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:52:10 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.70499999999999996,"15min":0.69599999999999995},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.863,"5min":6.0810000000000004,"15min":6.806},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.755,"15min":0.70799999999999996},"min":{"1min":0.23799999999999999,"5min":0.22900000000000001,"15min":0.17499999999999999},"max":{"1min":0.55200000000000005,"5min":6.734,"15min":7.9210000000000003},"last":0.79000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739244,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12505956,"kb_used_data":11646428,"kb_used_omap":1251,"kb_used_meta":858268,"kb_avail":81865884,"statfs":{"total":96636764160,"available":83830665216,"internally_reserved":0,"allocated":11925942272,"data_stored":32938539198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":878866625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,1,8,9,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.80800000000000005,"15min":0.78900000000000003},"min":{"1min":0.21099999999999999,"5min":0.20399999999999999,"15min":0.17799999999999999},"max":{"1min":7.0880000000000001,"5min":8.4030000000000005,"15min":11.715},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.0189999999999999,"15min":0.97999999999999998},"min":{"1min":0.215,"5min":0.183,"15min":0.183},"max":{"1min":7.0469999999999997,"5min":7.0469999999999997,"15min":12.058},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:52:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":0.68999999999999995,"15min":0.56699999999999995},"min":{"1min":0.245,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":8.5700000000000003,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.59299999999999997,"15min":0.56000000000000005},"min":{"1min":0.33200000000000002,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11923722240,"data_stored":32937295872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11924697088,"data_stored":32937308160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11924590592,"data_stored":32937254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:52:53.638 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:53.638 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:53.638 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:52:53.638 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:53:13.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:53:14.498 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:53:14.498 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:53:14.541 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2360,"stamp":"2023-09-23T03:53:14.272620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20554000916,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1176568,"num_read_kb":4138966,"num_write":1047890,"num_write_kb":4089233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85840,"ondisk_log_size":85840,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39804876,"kb_used_data":34981364,"kb_used_omap":3754,"kb_used_meta":4819733,"kb_avail":243310644,"statfs":{"total":289910292480,"available":249150099456,"internally_reserved":0,"allocated":35820916736,"data_stored":98815536806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4935407519},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,5,7,14,10,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7991296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1810,"num_read_kb":7220,"num_write":1959,"num_write_kb":7836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006393"},"pg_stats":[{"pgid":"2.7","version":"156'8003","reported_seq":13024,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.036891+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:53:08.036891+0000","last_peered":"2023-09-23T03:53:08.036891+0000","last_clean":"2023-09-23T03:53:08.036891+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:53:08.036891+0000","last_undegraded":"2023-09-23T03:53:08.036891+0000","last_fullsized":"2023-09-23T03:53:08.036891+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8003,"log_dups_size":0,"ondisk_log_size":8003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'5374","reported_seq":9270,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:07.708122+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:53:07.708122+0000","last_peered":"2023-09-23T03:53:07.708122+0000","last_clean":"2023-09-23T03:53:07.708122+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:53:07.708122+0000","last_undegraded":"2023-09-23T03:53:07.708122+0000","last_fullsized":"2023-09-23T03:53:07.708122+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5374,"log_dups_size":0,"ondisk_log_size":5374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'6978","reported_seq":11951,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.036283+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:53:08.036283+0000","last_peered":"2023-09-23T03:53:08.036283+0000","last_clean":"2023-09-23T03:53:08.036283+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:53:08.036283+0000","last_undegraded":"2023-09-23T03:53:08.036283+0000","last_fullsized":"2023-09-23T03:53:08.036283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'9657","reported_seq":17469,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.036430+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:53:08.036430+0000","last_peered":"2023-09-23T03:53:08.036430+0000","last_clean":"2023-09-23T03:53:08.036430+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:53:08.036430+0000","last_undegraded":"2023-09-23T03:53:08.036430+0000","last_fullsized":"2023-09-23T03:53:08.036430+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9657,"log_dups_size":0,"ondisk_log_size":9657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"156'255491","reported_seq":507955,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:10.738871+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:53:10.738871+0000","last_peered":"2023-09-23T03:53:10.738871+0000","last_clean":"2023-09-23T03:53:10.738871+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:53:10.738871+0000","last_undegraded":"2023-09-23T03:53:10.738871+0000","last_fullsized":"2023-09-23T03:53:10.738871+0000","mapping_epoch":18,"log_start":"153'245462","ondisk_log_start":"153'245462","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5122564096,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":254953,"num_read_kb":998800,"num_write":252467,"num_write_kb":1009868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2257,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:51:13.865746+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:51:13.865746+0000","last_peered":"2023-09-23T03:51:13.865746+0000","last_clean":"2023-09-23T03:51:13.865746+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:51:13.865746+0000","last_undegraded":"2023-09-23T03:51:13.865746+0000","last_fullsized":"2023-09-23T03:51:13.865746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"156'267125","reported_seq":531072,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:12.147345+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:53:12.147345+0000","last_peered":"2023-09-23T03:53:12.147345+0000","last_clean":"2023-09-23T03:53:12.147345+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:53:12.147345+0000","last_undegraded":"2023-09-23T03:53:12.147345+0000","last_fullsized":"2023-09-23T03:53:12.147345+0000","mapping_epoch":18,"log_start":"153'257119","ondisk_log_start":"153'257119","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3424,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 404s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5356802048,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":266596,"num_read_kb":1044088,"num_write":263944,"num_write_kb":1055776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5767,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.037432+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:53:08.037432+0000","last_peered":"2023-09-23T03:53:08.037432+0000","last_clean":"2023-09-23T03:53:08.037432+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:53:08.037432+0000","last_undegraded":"2023-09-23T03:53:08.037432+0000","last_fullsized":"2023-09-23T03:53:08.037432+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"156'260891","reported_seq":518703,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:10.709238+0000","last_change":"2023-09-23T03:46:15.991882+0000","last_active":"2023-09-23T03:53:10.709238+0000","last_peered":"2023-09-23T03:53:10.709238+0000","last_clean":"2023-09-23T03:53:10.709238+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:53:10.709238+0000","last_undegraded":"2023-09-23T03:53:10.709238+0000","last_fullsized":"2023-09-23T03:53:10.709238+0000","mapping_epoch":18,"log_start":"153'250847","ondisk_log_start":"153'250847","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3946,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 414s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5222191104,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":260344,"num_read_kb":1020652,"num_write":257807,"num_write_kb":1031228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4951,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.037386+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:53:08.037386+0000","last_peered":"2023-09-23T03:53:08.037386+0000","last_clean":"2023-09-23T03:53:08.037386+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:53:08.037386+0000","last_undegraded":"2023-09-23T03:53:08.037386+0000","last_fullsized":"2023-09-23T03:53:08.037386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"156'242277","reported_seq":481303,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:12.090521+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:53:12.090521+0000","last_peered":"2023-09-23T03:53:12.090521+0000","last_clean":"2023-09-23T03:53:12.090521+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:53:12.090521+0000","last_undegraded":"2023-09-23T03:53:12.090521+0000","last_fullsized":"2023-09-23T03:53:12.090521+0000","mapping_epoch":18,"log_start":"153'232248","ondisk_log_start":"153'232248","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1490,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 108s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4851838995,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":241560,"num_read_kb":947076,"num_write":239366,"num_write_kb":957458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"156'10249","reported_seq":18444,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:08.043508+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:53:08.043508+0000","last_peered":"2023-09-23T03:53:08.043508+0000","last_clean":"2023-09-23T03:53:08.043508+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:53:08.043508+0000","last_undegraded":"2023-09-23T03:53:08.043508+0000","last_fullsized":"2023-09-23T03:53:08.043508+0000","mapping_epoch":13,"log_start":"43'200","ondisk_log_start":"43'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":327,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-09-23T03:52:26.631129+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:26.631129+0000","last_peered":"2023-09-23T03:52:26.631129+0000","last_clean":"2023-09-23T03:52:26.631129+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:26.631129+0000","last_undegraded":"2023-09-23T03:52:26.631129+0000","last_fullsized":"2023-09-23T03:52:26.631129+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20553396243,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1023453,"num_read_kb":4010616,"num_write":1013584,"num_write_kb":4054330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35814031360,"data_stored":98812276736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40108,"ondisk_log_size":40108,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45633,"ondisk_log_size":45633,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739252,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13451652,"kb_used_data":11660284,"kb_used_omap":1245,"kb_used_meta":1790114,"kb_avail":80920188,"statfs":{"total":96636764160,"available":82862272512,"internally_reserved":0,"allocated":11940130816,"data_stored":32938113115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1833077276},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.88600000000000001,"15min":0.95899999999999996},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.3700000000000001,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.51200000000000001,"15min":0.56799999999999995},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.72199999999999998,"5min":13.618,"15min":13.618},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.2799999999999998,"5min":10.528,"15min":10.528},"last":2.2799999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.872,"15min":0.84299999999999997},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":3.1059999999999999,"5min":10.625,"15min":10.625},"last":0.61399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739250,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13761796,"kb_used_data":11661372,"kb_used_omap":1256,"kb_used_meta":2099159,"kb_avail":80610044,"statfs":{"total":96636764160,"available":82544685056,"internally_reserved":0,"allocated":11941244928,"data_stored":32938700173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":2149539010},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.72199999999999998,"15min":0.86599999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":2.6150000000000002,"5min":6.7039999999999997,"15min":7.0410000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.63400000000000001,"15min":0.73699999999999999},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":4.843,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.29599999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58299999999999996,"15min":0.69499999999999995},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.165,"5min":6.0810000000000004,"15min":6.806},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67500000000000004,"15min":0.71199999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":3.1970000000000001,"5min":6.0940000000000003,"15min":7.9210000000000003},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739248,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12591428,"kb_used_data":11659708,"kb_used_omap":1251,"kb_used_meta":930460,"kb_avail":81780412,"statfs":{"total":96636764160,"available":83743141888,"internally_reserved":0,"allocated":11939540992,"data_stored":32938723518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":952791233},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,5,9,7,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":1.165,"5min":8.4030000000000005,"15min":11.715},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":1.034,"15min":0.97399999999999998},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.183},"max":{"1min":3.8919999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.72499999999999998,"15min":0.57299999999999995},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":3.9180000000000001,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.58599999999999997,"15min":0.56499999999999995},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.96599999999999997,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11937415168,"data_stored":32937398272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11938365440,"data_stored":32937443328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11938250752,"data_stored":32937435136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:53:14.543 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:14.543 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:14.543 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:14.544 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:53:18.855 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.851+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:53:18.856 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:53:18.856 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.855+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:53:18.856 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.855+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:53:18.857 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.855+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:53:18.857 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:18.855+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:53:34.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:53:35.436 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:53:35.436 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:53:35.483 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2371,"stamp":"2023-09-23T03:53:34.283031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20570253844,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1180290,"num_read_kb":4153806,"num_write":1051819,"num_write_kb":4104949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86010,"ondisk_log_size":86010,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":40085332,"kb_used_data":35021808,"kb_used_omap":3754,"kb_used_meta":5059733,"kb_avail":243030188,"statfs":{"total":289910292480,"available":248862912512,"internally_reserved":0,"allocated":35862331392,"data_stored":98816236051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":5181167519},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,8,12,10,8,17,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8110080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1889,"num_read_kb":7532,"num_write":1980,"num_write_kb":7920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006616"},"pg_stats":[{"pgid":"2.7","version":"157'8021","reported_seq":13055,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:32.001957+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:53:32.001957+0000","last_peered":"2023-09-23T03:53:32.001957+0000","last_clean":"2023-09-23T03:53:32.001957+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:53:32.001957+0000","last_undegraded":"2023-09-23T03:53:32.001957+0000","last_fullsized":"2023-09-23T03:53:32.001957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8021,"log_dups_size":0,"ondisk_log_size":8021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'5380","reported_seq":9281,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.896576+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:53:28.896576+0000","last_peered":"2023-09-23T03:53:28.896576+0000","last_clean":"2023-09-23T03:53:28.896576+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:53:28.896576+0000","last_undegraded":"2023-09-23T03:53:28.896576+0000","last_fullsized":"2023-09-23T03:53:28.896576+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5380,"log_dups_size":0,"ondisk_log_size":5380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'6996","reported_seq":11983,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:32.001106+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:53:32.001106+0000","last_peered":"2023-09-23T03:53:32.001106+0000","last_clean":"2023-09-23T03:53:32.001106+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:53:32.001106+0000","last_undegraded":"2023-09-23T03:53:32.001106+0000","last_fullsized":"2023-09-23T03:53:32.001106+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'9680","reported_seq":17510,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:32.015071+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:53:32.015071+0000","last_peered":"2023-09-23T03:53:32.015071+0000","last_clean":"2023-09-23T03:53:32.015071+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:53:32.015071+0000","last_undegraded":"2023-09-23T03:53:32.015071+0000","last_fullsized":"2023-09-23T03:53:32.015071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9680,"log_dups_size":0,"ondisk_log_size":9680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'256474","reported_seq":509852,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:30.973738+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:53:30.973738+0000","last_peered":"2023-09-23T03:53:30.973738+0000","last_clean":"2023-09-23T03:53:30.973738+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:53:30.973738+0000","last_undegraded":"2023-09-23T03:53:30.973738+0000","last_fullsized":"2023-09-23T03:53:30.973738+0000","mapping_epoch":18,"log_start":"154'246468","ondisk_log_start":"154'246468","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5126717440,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":255865,"num_read_kb":1002436,"num_write":253450,"num_write_kb":1013800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2259,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:15.839384+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:15.839384+0000","last_peered":"2023-09-23T03:53:15.839384+0000","last_clean":"2023-09-23T03:53:15.839384+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:15.839384+0000","last_undegraded":"2023-09-23T03:53:15.839384+0000","last_fullsized":"2023-09-23T03:53:15.839384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'268114","reported_seq":533001,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:32.290436+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:53:32.290436+0000","last_peered":"2023-09-23T03:53:32.290436+0000","last_clean":"2023-09-23T03:53:32.290436+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:53:32.290436+0000","last_undegraded":"2023-09-23T03:53:32.290436+0000","last_fullsized":"2023-09-23T03:53:32.290436+0000","mapping_epoch":18,"log_start":"154'258037","ondisk_log_start":"154'258037","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3571,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 425s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5360889856,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":267527,"num_read_kb":1047804,"num_write":264933,"num_write_kb":1059732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5776,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.516545+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:53:28.516545+0000","last_peered":"2023-09-23T03:53:28.516545+0000","last_clean":"2023-09-23T03:53:28.516545+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:53:28.516545+0000","last_undegraded":"2023-09-23T03:53:28.516545+0000","last_fullsized":"2023-09-23T03:53:28.516545+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'261902","reported_seq":520721,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:30.997083+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:53:30.997083+0000","last_peered":"2023-09-23T03:53:30.997083+0000","last_clean":"2023-09-23T03:53:30.997083+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:53:30.997083+0000","last_undegraded":"2023-09-23T03:53:30.997083+0000","last_fullsized":"2023-09-23T03:53:30.997083+0000","mapping_epoch":18,"log_start":"154'251856","ondisk_log_start":"154'251856","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5226332160,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":261332,"num_read_kb":1024600,"num_write":258818,"num_write_kb":1035272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4960,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.539888+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:53:28.539888+0000","last_peered":"2023-09-23T03:53:28.539888+0000","last_clean":"2023-09-23T03:53:28.539888+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:53:28.539888+0000","last_undegraded":"2023-09-23T03:53:28.539888+0000","last_fullsized":"2023-09-23T03:53:28.539888+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'243223","reported_seq":483157,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:32.342200+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:53:32.342200+0000","last_peered":"2023-09-23T03:53:32.342200+0000","last_clean":"2023-09-23T03:53:32.342200+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:53:32.342200+0000","last_undegraded":"2023-09-23T03:53:32.342200+0000","last_fullsized":"2023-09-23T03:53:32.342200+0000","mapping_epoch":18,"log_start":"153'233164","ondisk_log_start":"153'233164","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":1757,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 129s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4855709715,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":242451,"num_read_kb":950616,"num_write":240312,"num_write_kb":961242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'10274","reported_seq":18491,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:32.070216+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:53:32.070216+0000","last_peered":"2023-09-23T03:53:32.070216+0000","last_clean":"2023-09-23T03:53:32.070216+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:53:32.070216+0000","last_undegraded":"2023-09-23T03:53:32.070216+0000","last_fullsized":"2023-09-23T03:53:32.070216+0000","mapping_epoch":13,"log_start":"43'200","ondisk_log_start":"43'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":328,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:28.900223+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:28.900223+0000","last_peered":"2023-09-23T03:52:28.900223+0000","last_clean":"2023-09-23T03:52:28.900223+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:28.900223+0000","last_undegraded":"2023-09-23T03:52:28.900223+0000","last_fullsized":"2023-09-23T03:52:28.900223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20569649171,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1027175,"num_read_kb":4025456,"num_write":1017513,"num_write_kb":4070046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35855171584,"data_stored":98813001728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40188,"ondisk_log_size":40188,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45723,"ondisk_log_size":45723,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739256,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13555132,"kb_used_data":11673524,"kb_used_omap":1245,"kb_used_meta":1880354,"kb_avail":80816708,"statfs":{"total":96636764160,"available":82756308992,"internally_reserved":0,"allocated":11953688576,"data_stored":32938331178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1925483036},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.88600000000000001,"15min":0.95899999999999996},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.3700000000000001,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.51200000000000001,"15min":0.56799999999999995},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.72199999999999998,"5min":13.618,"15min":13.618},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.2799999999999998,"5min":10.528,"15min":10.528},"last":0.441},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.872,"15min":0.84299999999999997},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":3.1059999999999999,"5min":10.625,"15min":10.625},"last":0.46600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739254,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13870640,"kb_used_data":11674600,"kb_used_omap":1256,"kb_used_meta":2194775,"kb_avail":80501200,"statfs":{"total":96636764160,"available":82433228800,"internally_reserved":0,"allocated":11954790400,"data_stored":32938951004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":2247449794},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,4,6,17,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.72199999999999998,"15min":0.86599999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":2.6150000000000002,"5min":6.7039999999999997,"15min":7.0410000000000004},"last":0.372},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.63400000000000001,"15min":0.73699999999999999},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":4.843,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58299999999999996,"15min":0.69499999999999995},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.165,"5min":6.0810000000000004,"15min":6.806},"last":0.316},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67500000000000004,"15min":0.71199999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":3.1970000000000001,"5min":6.0940000000000003,"15min":7.9210000000000003},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739252,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12659560,"kb_used_data":11673684,"kb_used_omap":1251,"kb_used_meta":984604,"kb_avail":81712280,"statfs":{"total":96636764160,"available":83673374720,"internally_reserved":0,"allocated":11953852416,"data_stored":32938953869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1008234689},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,9,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":1.165,"5min":8.4030000000000005,"15min":11.715},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":1.034,"15min":0.97399999999999998},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.183},"max":{"1min":3.8919999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.72499999999999998,"15min":0.57299999999999995},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":3.9180000000000001,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.58599999999999997,"15min":0.56499999999999995},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.96599999999999997,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11951226880,"data_stored":32937672704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11952082944,"data_stored":32937705472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11951861760,"data_stored":32937623552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:53:35.485 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:35.485 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:35.486 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:53:35.698 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.698+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:53:35.711 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:53:35.712 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:53:35.712 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:53:35.710+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:53:55.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:53:56.190 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:53:56.191 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:53:56.249 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2381,"stamp":"2023-09-23T03:53:54.287056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20586113556,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1183985,"num_read_kb":4168534,"num_write":1055680,"num_write_kb":4120393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86195,"ondisk_log_size":86195,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39149248,"kb_used_data":35061160,"kb_used_omap":3754,"kb_used_meta":4084309,"kb_avail":243966272,"statfs":{"total":289910292480,"available":249821462528,"internally_reserved":0,"allocated":35902627840,"data_stored":98816719379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4182333343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,9,5,7,16,20,9,14,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10072064,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2278,"num_read_kb":9072,"num_write":2462,"num_write_kb":9848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002201"},"pg_stats":[{"pgid":"2.7","version":"157'8036","reported_seq":13082,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:52.367616+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:53:52.367616+0000","last_peered":"2023-09-23T03:53:52.367616+0000","last_clean":"2023-09-23T03:53:52.367616+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:53:52.367616+0000","last_undegraded":"2023-09-23T03:53:52.367616+0000","last_fullsized":"2023-09-23T03:53:52.367616+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8036,"log_dups_size":0,"ondisk_log_size":8036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'5385","reported_seq":9290,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:50.675847+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:53:50.675847+0000","last_peered":"2023-09-23T03:53:50.675847+0000","last_clean":"2023-09-23T03:53:50.675847+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:53:50.675847+0000","last_undegraded":"2023-09-23T03:53:50.675847+0000","last_fullsized":"2023-09-23T03:53:50.675847+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5385,"log_dups_size":0,"ondisk_log_size":5385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'7011","reported_seq":12010,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:52.365713+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:53:52.365713+0000","last_peered":"2023-09-23T03:53:52.365713+0000","last_clean":"2023-09-23T03:53:52.365713+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:53:52.365713+0000","last_undegraded":"2023-09-23T03:53:52.365713+0000","last_fullsized":"2023-09-23T03:53:52.365713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'9700","reported_seq":17546,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:52.391484+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:53:52.391484+0000","last_peered":"2023-09-23T03:53:52.391484+0000","last_clean":"2023-09-23T03:53:52.391484+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:53:52.391484+0000","last_undegraded":"2023-09-23T03:53:52.391484+0000","last_fullsized":"2023-09-23T03:53:52.391484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'257440","reported_seq":511728,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:51.130599+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:53:51.130599+0000","last_peered":"2023-09-23T03:53:51.130599+0000","last_clean":"2023-09-23T03:53:51.130599+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:53:51.130599+0000","last_undegraded":"2023-09-23T03:53:51.130599+0000","last_fullsized":"2023-09-23T03:53:51.130599+0000","mapping_epoch":18,"log_start":"154'247374","ondisk_log_start":"154'247374","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5130711040,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":256780,"num_read_kb":1006088,"num_write":254416,"num_write_kb":1017664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2259,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:15.839384+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:15.839384+0000","last_peered":"2023-09-23T03:53:15.839384+0000","last_clean":"2023-09-23T03:53:15.839384+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:15.839384+0000","last_undegraded":"2023-09-23T03:53:15.839384+0000","last_fullsized":"2023-09-23T03:53:15.839384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'269122","reported_seq":534968,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:52.604466+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:53:52.604466+0000","last_peered":"2023-09-23T03:53:52.604466+0000","last_clean":"2023-09-23T03:53:52.604466+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:53:52.604466+0000","last_undegraded":"2023-09-23T03:53:52.604466+0000","last_fullsized":"2023-09-23T03:53:52.604466+0000","mapping_epoch":18,"log_start":"154'259065","ondisk_log_start":"154'259065","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3701,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 445s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5365014528,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":268468,"num_read_kb":1051548,"num_write":265941,"num_write_kb":1063764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5784,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:49.022377+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:53:49.022377+0000","last_peered":"2023-09-23T03:53:49.022377+0000","last_clean":"2023-09-23T03:53:49.022377+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:53:49.022377+0000","last_undegraded":"2023-09-23T03:53:49.022377+0000","last_fullsized":"2023-09-23T03:53:49.022377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'262865","reported_seq":522641,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:51.131087+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:53:51.131087+0000","last_peered":"2023-09-23T03:53:51.131087+0000","last_clean":"2023-09-23T03:53:51.131087+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:53:51.131087+0000","last_undegraded":"2023-09-23T03:53:51.131087+0000","last_fullsized":"2023-09-23T03:53:51.131087+0000","mapping_epoch":18,"log_start":"154'252768","ondisk_log_start":"154'252768","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5230301184,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":262286,"num_read_kb":1028400,"num_write":259781,"num_write_kb":1039124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4968,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:49.076859+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:53:49.076859+0000","last_peered":"2023-09-23T03:53:49.076859+0000","last_clean":"2023-09-23T03:53:49.076859+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:53:49.076859+0000","last_undegraded":"2023-09-23T03:53:49.076859+0000","last_fullsized":"2023-09-23T03:53:49.076859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'244147","reported_seq":484978,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:53:52.606082+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:53:52.606082+0000","last_peered":"2023-09-23T03:53:52.606082+0000","last_clean":"2023-09-23T03:53:52.606082+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:53:52.606082+0000","last_undegraded":"2023-09-23T03:53:52.606082+0000","last_fullsized":"2023-09-23T03:53:52.606082+0000","mapping_epoch":18,"log_start":"154'234073","ondisk_log_start":"154'234073","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2032,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 149s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4859482131,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":243336,"num_read_kb":954148,"num_write":241236,"num_write_kb":964938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'10298","reported_seq":18535,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:52.453679+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:53:52.453679+0000","last_peered":"2023-09-23T03:53:52.453679+0000","last_clean":"2023-09-23T03:53:52.453679+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:53:52.453679+0000","last_undegraded":"2023-09-23T03:53:52.453679+0000","last_fullsized":"2023-09-23T03:53:52.453679+0000","mapping_epoch":13,"log_start":"43'200","ondisk_log_start":"43'200","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":328,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:28.900223+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:28.900223+0000","last_peered":"2023-09-23T03:52:28.900223+0000","last_clean":"2023-09-23T03:52:28.900223+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:28.900223+0000","last_undegraded":"2023-09-23T03:52:28.900223+0000","last_fullsized":"2023-09-23T03:52:28.900223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20585508883,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1030870,"num_read_kb":4040184,"num_write":1021374,"num_write_kb":4085490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35895754752,"data_stored":98813415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40294,"ondisk_log_size":40294,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45802,"ondisk_log_size":45802,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739260,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13622812,"kb_used_data":11686676,"kb_used_omap":1245,"kb_used_meta":1934882,"kb_avail":80749028,"statfs":{"total":96636764160,"available":82687004672,"internally_reserved":0,"allocated":11967156224,"data_stored":32938507306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1981319708},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.88600000000000001,"15min":0.95899999999999996},"min":{"1min":0.20999999999999999,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":3.3700000000000001,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.51200000000000001,"15min":0.56799999999999995},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":0.72199999999999998,"5min":13.618,"15min":13.618},"last":0.25}]},{"osd":1,"last update":"Sat Sep 23 03:53:06 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55700000000000005,"15min":0.58999999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.2799999999999998,"5min":10.528,"15min":10.528},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.872,"15min":0.84299999999999997},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":3.1059999999999999,"5min":10.625,"15min":10.625},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359739258,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12799748,"kb_used_data":11687804,"kb_used_omap":1256,"kb_used_meta":1110679,"kb_avail":81572092,"statfs":{"total":96636764160,"available":83529822208,"internally_reserved":0,"allocated":11968311296,"data_stored":32939118940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1137335490},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,10,11,7,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.72199999999999998,"15min":0.86599999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":2.6150000000000002,"5min":6.7039999999999997,"15min":7.0410000000000004},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.63400000000000001,"15min":0.73699999999999999},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.185},"max":{"1min":4.843,"5min":6.0529999999999999,"15min":7.2569999999999997},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:53:10 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.58299999999999996,"15min":0.69499999999999995},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.165,"5min":6.0810000000000004,"15min":6.806},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.67500000000000004,"15min":0.71199999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":3.1970000000000001,"5min":6.0940000000000003,"15min":7.9210000000000003},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739256,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12726688,"kb_used_data":11686680,"kb_used_omap":1251,"kb_used_meta":1038748,"kb_avail":81645152,"statfs":{"total":96636764160,"available":83604635648,"internally_reserved":0,"allocated":11967160320,"data_stored":32939093133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1063678145},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,9,3,5,5,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":1.165,"5min":8.4030000000000005,"15min":11.715},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":1.034,"15min":0.97399999999999998},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.183},"max":{"1min":3.8919999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.223}]},{"osd":2,"last update":"Sat Sep 23 03:53:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.72499999999999998,"15min":0.57299999999999995},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.20100000000000001},"max":{"1min":3.9180000000000001,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.58599999999999997,"15min":0.56499999999999995},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.96599999999999997,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11964649472,"data_stored":32937824256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11965681664,"data_stored":32937779200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11965423616,"data_stored":32937811968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:53:56.250 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:56.251 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:53:56.251 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:54:16.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:54:17.314 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:54:17.315 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:54:17.362 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2392,"stamp":"2023-09-23T03:54:16.290814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20602309140,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1187980,"num_read_kb":4184442,"num_write":1059615,"num_write_kb":4136133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86127,"ondisk_log_size":86127,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39475652,"kb_used_data":35101356,"kb_used_omap":3754,"kb_used_meta":4370517,"kb_avail":243639868,"statfs":{"total":289910292480,"available":249487224832,"internally_reserved":0,"allocated":35943788544,"data_stored":98817247763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4475410335},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,4,4,0,11,15,10,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7966720,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1925,"num_read_kb":7656,"num_write":1943,"num_write_kb":7772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002026"},"pg_stats":[{"pgid":"2.7","version":"157'8048","reported_seq":13107,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:12.311036+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:54:12.311036+0000","last_peered":"2023-09-23T03:54:12.311036+0000","last_clean":"2023-09-23T03:54:12.311036+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:54:12.311036+0000","last_undegraded":"2023-09-23T03:54:12.311036+0000","last_fullsized":"2023-09-23T03:54:12.311036+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8048,"log_dups_size":0,"ondisk_log_size":8048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'5389","reported_seq":9298,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:10.753131+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:54:10.753131+0000","last_peered":"2023-09-23T03:54:10.753131+0000","last_clean":"2023-09-23T03:54:10.753131+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:54:10.753131+0000","last_undegraded":"2023-09-23T03:54:10.753131+0000","last_fullsized":"2023-09-23T03:54:10.753131+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'7023","reported_seq":12034,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:12.311080+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:54:12.311080+0000","last_peered":"2023-09-23T03:54:12.311080+0000","last_clean":"2023-09-23T03:54:12.311080+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:54:12.311080+0000","last_undegraded":"2023-09-23T03:54:12.311080+0000","last_fullsized":"2023-09-23T03:54:12.311080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7023,"log_dups_size":0,"ondisk_log_size":7023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'9716","reported_seq":17578,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:12.327132+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:54:12.327132+0000","last_peered":"2023-09-23T03:54:12.327132+0000","last_clean":"2023-09-23T03:54:12.327132+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:54:12.327132+0000","last_undegraded":"2023-09-23T03:54:12.327132+0000","last_fullsized":"2023-09-23T03:54:12.327132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'258376","reported_seq":513647,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:11.385752+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:54:11.385752+0000","last_peered":"2023-09-23T03:54:11.385752+0000","last_clean":"2023-09-23T03:54:11.385752+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:54:11.385752+0000","last_undegraded":"2023-09-23T03:54:11.385752+0000","last_fullsized":"2023-09-23T03:54:11.385752+0000","mapping_epoch":18,"log_start":"154'248289","ondisk_log_start":"154'248289","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5134573568,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":257759,"num_read_kb":1009988,"num_write":255352,"num_write_kb":1021408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2259,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:15.839384+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:15.839384+0000","last_peered":"2023-09-23T03:53:15.839384+0000","last_clean":"2023-09-23T03:53:15.839384+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:15.839384+0000","last_undegraded":"2023-09-23T03:53:15.839384+0000","last_fullsized":"2023-09-23T03:53:15.839384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'270146","reported_seq":537012,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:54:12.849547+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:54:12.849547+0000","last_peered":"2023-09-23T03:54:12.849547+0000","last_clean":"2023-09-23T03:54:12.849547+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:54:12.849547+0000","last_undegraded":"2023-09-23T03:54:12.849547+0000","last_fullsized":"2023-09-23T03:54:12.849547+0000","mapping_epoch":18,"log_start":"154'260090","ondisk_log_start":"154'260090","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3895,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 465s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5369208832,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":269474,"num_read_kb":1055552,"num_write":266965,"num_write_kb":1067860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5792,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:09.510531+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:54:09.510531+0000","last_peered":"2023-09-23T03:54:09.510531+0000","last_clean":"2023-09-23T03:54:09.510531+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:54:09.510531+0000","last_undegraded":"2023-09-23T03:54:09.510531+0000","last_fullsized":"2023-09-23T03:54:09.510531+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'263918","reported_seq":524746,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:11.379875+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:54:11.379875+0000","last_peered":"2023-09-23T03:54:11.379875+0000","last_clean":"2023-09-23T03:54:11.379875+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:54:11.379875+0000","last_undegraded":"2023-09-23T03:54:11.379875+0000","last_fullsized":"2023-09-23T03:54:11.379875+0000","mapping_epoch":18,"log_start":"154'253883","ondisk_log_start":"154'253883","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5234659328,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":263337,"num_read_kb":1032580,"num_write":260834,"num_write_kb":1043336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4976,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:09.563729+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:54:09.563729+0000","last_peered":"2023-09-23T03:54:09.563729+0000","last_clean":"2023-09-23T03:54:09.563729+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:54:09.563729+0000","last_undegraded":"2023-09-23T03:54:09.563729+0000","last_fullsized":"2023-09-23T03:54:09.563729+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'245069","reported_seq":486881,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:54:12.876993+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:54:12.876993+0000","last_peered":"2023-09-23T03:54:12.876993+0000","last_clean":"2023-09-23T03:54:12.876993+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:54:12.876993+0000","last_undegraded":"2023-09-23T03:54:12.876993+0000","last_fullsized":"2023-09-23T03:54:12.876993+0000","mapping_epoch":18,"log_start":"154'234985","ondisk_log_start":"154'234985","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2377,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 169s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4863262739,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":244295,"num_read_kb":957972,"num_write":242158,"num_write_kb":968626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'10318","reported_seq":18575,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:12.373363+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:54:12.373363+0000","last_peered":"2023-09-23T03:54:12.373363+0000","last_clean":"2023-09-23T03:54:12.373363+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:54:12.373363+0000","last_undegraded":"2023-09-23T03:54:12.373363+0000","last_fullsized":"2023-09-23T03:54:12.373363+0000","mapping_epoch":13,"log_start":"43'300","ondisk_log_start":"43'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":328,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:52:28.900223+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:52:28.900223+0000","last_peered":"2023-09-23T03:52:28.900223+0000","last_clean":"2023-09-23T03:52:28.900223+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:52:28.900223+0000","last_undegraded":"2023-09-23T03:52:28.900223+0000","last_fullsized":"2023-09-23T03:52:28.900223+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20601704467,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1034865,"num_read_kb":4056092,"num_write":1025309,"num_write_kb":4101230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35937140736,"data_stored":98813841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40262,"ondisk_log_size":40262,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45766,"ondisk_log_size":45766,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739264,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13743096,"kb_used_data":11700144,"kb_used_omap":1245,"kb_used_meta":2041698,"kb_avail":80628744,"statfs":{"total":96636764160,"available":82563833856,"internally_reserved":0,"allocated":11980947456,"data_stored":32938687530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":2090699292},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.86899999999999999,"15min":0.96599999999999997},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":4.21,"5min":14.617000000000001,"15min":14.617000000000001},"last":2.262},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.25,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":4.1150000000000002,"5min":13.618,"15min":13.618},"last":2.2879999999999998}]},{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.57999999999999996,"15min":0.60099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":1.3779999999999999,"5min":10.528,"15min":10.528},"last":2.5110000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.85499999999999998,"15min":0.83099999999999996},"min":{"1min":0.30299999999999999,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":1.413,"5min":10.625,"15min":10.625},"last":2.4710000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739262,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12867136,"kb_used_data":11701048,"kb_used_omap":1256,"kb_used_meta":1164823,"kb_avail":81504704,"statfs":{"total":96636764160,"available":83460816896,"internally_reserved":0,"allocated":11981873152,"data_stored":32939286876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1192778946},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,0,0,0,3,0,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":33.228999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.8129999999999999,"5min":0.90300000000000002,"15min":0.80400000000000005},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.77000000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":0.85799999999999998,"15min":0.76700000000000002},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":33.505000000000003,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.839,"5min":0.89200000000000002,"15min":0.79700000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":33.408999999999999,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.48999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739260,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12865420,"kb_used_data":11700164,"kb_used_omap":1251,"kb_used_meta":1163996,"kb_avail":81506420,"statfs":{"total":96636764160,"available":83462574080,"internally_reserved":0,"allocated":11980967936,"data_stored":32939273357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1191932097},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,4,0,5,15,4,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.64400000000000002,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":0.56100000000000005,"5min":7.0880000000000001,"15min":11.715},"last":3.0470000000000002},{"interface":"front","average":{"1min":0.27000000000000002,"5min":0.874,"15min":0.89600000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":7.0469999999999997,"15min":12.058},"last":3.0139999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.66700000000000004,"15min":0.56399999999999995},"min":{"1min":0.219,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.54300000000000004,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53200000000000003,"15min":0.55100000000000005},"min":{"1min":0.25,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57699999999999996,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.38100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11978522624,"data_stored":32937938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11979522048,"data_stored":32937967616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11979096064,"data_stored":32937934848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:54:17.364 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:54:17.364 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:54:17.365 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:54:23.019 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.013+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.013+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.013+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:54:23.020 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:54:23.021 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:54:23.021 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:54:23.021 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:54:23.021 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:54:23.022 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:23.017+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:54:37.366 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:54:38.305 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:54:38.306 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:54:38.368 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2403,"stamp":"2023-09-23T03:54:38.299659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20622756372,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1192908,"num_read_kb":4204078,"num_write":1064535,"num_write_kb":4155813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86191,"ondisk_log_size":86191,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38540196,"kb_used_data":35153356,"kb_used_omap":3754,"kb_used_meta":3383061,"kb_avail":244575324,"statfs":{"total":289910292480,"available":250445131776,"internally_reserved":0,"allocated":35997036544,"data_stored":98818103827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":3464255391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,12,16,19,22,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12169216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2924,"num_read_kb":11648,"num_write":2909,"num_write_kb":11636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007045"},"pg_stats":[{"pgid":"2.7","version":"157'8072","reported_seq":13146,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.500615+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:54:36.500615+0000","last_peered":"2023-09-23T03:54:36.500615+0000","last_clean":"2023-09-23T03:54:36.500615+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:54:36.500615+0000","last_undegraded":"2023-09-23T03:54:36.500615+0000","last_fullsized":"2023-09-23T03:54:36.500615+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'5397","reported_seq":9312,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:35.076567+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:54:35.076567+0000","last_peered":"2023-09-23T03:54:35.076567+0000","last_clean":"2023-09-23T03:54:35.076567+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:54:35.076567+0000","last_undegraded":"2023-09-23T03:54:35.076567+0000","last_fullsized":"2023-09-23T03:54:35.076567+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'7047","reported_seq":12073,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.489379+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:54:36.489379+0000","last_peered":"2023-09-23T03:54:36.489379+0000","last_clean":"2023-09-23T03:54:36.489379+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:54:36.489379+0000","last_undegraded":"2023-09-23T03:54:36.489379+0000","last_fullsized":"2023-09-23T03:54:36.489379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7047,"log_dups_size":0,"ondisk_log_size":7047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'9748","reported_seq":17630,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.510147+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:54:36.510147+0000","last_peered":"2023-09-23T03:54:36.510147+0000","last_clean":"2023-09-23T03:54:36.510147+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:54:36.510147+0000","last_undegraded":"2023-09-23T03:54:36.510147+0000","last_fullsized":"2023-09-23T03:54:36.510147+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"157'259641","reported_seq":516143,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.644616+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:54:36.644616+0000","last_peered":"2023-09-23T03:54:36.644616+0000","last_clean":"2023-09-23T03:54:36.644616+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:54:36.644616+0000","last_undegraded":"2023-09-23T03:54:36.644616+0000","last_fullsized":"2023-09-23T03:54:36.644616+0000","mapping_epoch":18,"log_start":"154'249612","ondisk_log_start":"154'249612","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5139836928,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":258990,"num_read_kb":1014880,"num_write":256617,"num_write_kb":1026468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2259,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-09-23T03:53:15.839384+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:15.839384+0000","last_peered":"2023-09-23T03:53:15.839384+0000","last_clean":"2023-09-23T03:53:15.839384+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:15.839384+0000","last_undegraded":"2023-09-23T03:53:15.839384+0000","last_fullsized":"2023-09-23T03:53:15.839384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"157'271382","reported_seq":539509,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:54:38.224067+0000","last_change":"2023-09-23T03:46:27.268403+0000","last_active":"2023-09-23T03:54:38.224067+0000","last_peered":"2023-09-23T03:54:38.224067+0000","last_clean":"2023-09-23T03:54:38.224067+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:54:38.224067+0000","last_undegraded":"2023-09-23T03:54:38.224067+0000","last_fullsized":"2023-09-23T03:54:38.224067+0000","mapping_epoch":18,"log_start":"155'261318","ondisk_log_start":"155'261318","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":4158,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 490s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5374341120,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":270726,"num_read_kb":1060544,"num_write":268201,"num_write_kb":1072804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5802,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:35.077173+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:54:35.077173+0000","last_peered":"2023-09-23T03:54:35.077173+0000","last_clean":"2023-09-23T03:54:35.077173+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:54:35.077173+0000","last_undegraded":"2023-09-23T03:54:35.077173+0000","last_fullsized":"2023-09-23T03:54:35.077173+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"157'265187","reported_seq":527241,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.603242+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:54:36.603242+0000","last_peered":"2023-09-23T03:54:36.603242+0000","last_clean":"2023-09-23T03:54:36.603242+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:54:36.603242+0000","last_undegraded":"2023-09-23T03:54:36.603242+0000","last_fullsized":"2023-09-23T03:54:36.603242+0000","mapping_epoch":18,"log_start":"155'255097","ondisk_log_start":"155'255097","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5239963648,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":264565,"num_read_kb":1037484,"num_write":262103,"num_write_kb":1048412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4987,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:35.083584+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:54:35.083584+0000","last_peered":"2023-09-23T03:54:35.083584+0000","last_clean":"2023-09-23T03:54:35.083584+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:54:35.083584+0000","last_undegraded":"2023-09-23T03:54:35.083584+0000","last_fullsized":"2023-09-23T03:54:35.083584+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"157'246219","reported_seq":489263,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:54:38.224197+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:54:38.224197+0000","last_peered":"2023-09-23T03:54:38.224197+0000","last_clean":"2023-09-23T03:54:38.224197+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:54:38.224197+0000","last_undegraded":"2023-09-23T03:54:38.224197+0000","last_fullsized":"2023-09-23T03:54:38.224197+0000","mapping_epoch":18,"log_start":"154'236204","ondisk_log_start":"154'236204","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2640,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 195s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4868010003,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":245512,"num_read_kb":962820,"num_write":243308,"num_write_kb":973226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'10358","reported_seq":18640,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:36.579319+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:54:36.579319+0000","last_peered":"2023-09-23T03:54:36.579319+0000","last_clean":"2023-09-23T03:54:36.579319+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:54:36.579319+0000","last_undegraded":"2023-09-23T03:54:36.579319+0000","last_fullsized":"2023-09-23T03:54:36.579319+0000","mapping_epoch":13,"log_start":"43'300","ondisk_log_start":"43'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":329,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.986244+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:53:28.986244+0000","last_peered":"2023-09-23T03:53:28.986244+0000","last_clean":"2023-09-23T03:53:28.986244+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:53:28.986244+0000","last_undegraded":"2023-09-23T03:53:28.986244+0000","last_fullsized":"2023-09-23T03:53:28.986244+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20622151699,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1039793,"num_read_kb":4075728,"num_write":1030229,"num_write_kb":4120910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":35989823488,"data_stored":98814767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40198,"ondisk_log_size":40198,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45894,"ondisk_log_size":45894,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739269,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12862588,"kb_used_data":11717172,"kb_used_omap":1245,"kb_used_meta":1144162,"kb_avail":81509252,"statfs":{"total":96636764160,"available":83465474048,"internally_reserved":0,"allocated":11998384128,"data_stored":32938974250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1171622428},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.86899999999999999,"15min":0.96599999999999997},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":4.21,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.25,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":4.1150000000000002,"5min":13.618,"15min":13.618},"last":0.315}]},{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.57999999999999996,"15min":0.60099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":1.3779999999999999,"5min":10.528,"15min":10.528},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.85499999999999998,"15min":0.83099999999999996},"min":{"1min":0.30299999999999999,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":1.413,"5min":10.625,"15min":10.625},"last":0.38800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739267,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12751832,"kb_used_data":11718800,"kb_used_omap":1256,"kb_used_meta":1031767,"kb_avail":81620008,"statfs":{"total":96636764160,"available":83578888192,"internally_reserved":0,"allocated":12000051200,"data_stored":32939569500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1056529602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,11,13,12,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":33.228999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.8129999999999999,"5min":0.90300000000000002,"15min":0.80400000000000005},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":0.85799999999999998,"15min":0.76700000000000002},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":33.505000000000003,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.31},{"interface":"front","average":{"1min":1.839,"5min":0.89200000000000002,"15min":0.79700000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":33.408999999999999,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.40600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739265,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12925776,"kb_used_data":11717384,"kb_used_omap":1251,"kb_used_meta":1207132,"kb_avail":81446064,"statfs":{"total":96636764160,"available":83400769536,"internally_reserved":0,"allocated":11998601216,"data_stored":32939560077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1236103361},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,8,4,6,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.64400000000000002,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":0.56100000000000005,"5min":7.0880000000000001,"15min":11.715},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.27000000000000002,"5min":0.874,"15min":0.89600000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":7.0469999999999997,"15min":12.058},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.66700000000000004,"15min":0.56399999999999995},"min":{"1min":0.219,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.54300000000000004,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53200000000000003,"15min":0.55100000000000005},"min":{"1min":0.25,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57699999999999996,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11996114944,"data_stored":32938266624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11997097984,"data_stored":32938283008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11996610560,"data_stored":32938217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:54:38.370 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:54:38.370 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:54:38.370 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:54:57.798 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.796+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:54:57.798 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.796+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:54:57.798 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.796+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:54:57.798 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.796+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:54:57.807 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.800+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:54:57.807 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.800+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:54:57.808 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:57.800+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:54:58.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:54:59.122 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:54:59.123 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:54:59.183 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2414,"stamp":"2023-09-23T03:54:58.306442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20637809172,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1196641,"num_read_kb":4218918,"num_write":1068171,"num_write_kb":4170357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86246,"ondisk_log_size":86246,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38645120,"kb_used_data":35192552,"kb_used_omap":3754,"kb_used_meta":3448789,"kb_avail":244470400,"statfs":{"total":289910292480,"available":250337689600,"internally_reserved":0,"allocated":36037173248,"data_stored":98818635007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":3531560863},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,6,10,13,14,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11063296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2692,"num_read_kb":10704,"num_write":2654,"num_write_kb":10616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002064"},"pg_stats":[{"pgid":"2.7","version":"158'8084","reported_seq":13168,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:51.024773+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:54:51.024773+0000","last_peered":"2023-09-23T03:54:51.024773+0000","last_clean":"2023-09-23T03:54:51.024773+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:54:51.024773+0000","last_undegraded":"2023-09-23T03:54:51.024773+0000","last_fullsized":"2023-09-23T03:54:51.024773+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8084,"log_dups_size":0,"ondisk_log_size":8084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'5401","reported_seq":9320,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:50.438324+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:54:50.438324+0000","last_peered":"2023-09-23T03:54:50.438324+0000","last_clean":"2023-09-23T03:54:50.438324+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:54:50.438324+0000","last_undegraded":"2023-09-23T03:54:50.438324+0000","last_fullsized":"2023-09-23T03:54:50.438324+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'7059","reported_seq":12095,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:51.021309+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:54:51.021309+0000","last_peered":"2023-09-23T03:54:51.021309+0000","last_clean":"2023-09-23T03:54:51.021309+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:54:51.021309+0000","last_undegraded":"2023-09-23T03:54:51.021309+0000","last_fullsized":"2023-09-23T03:54:51.021309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7059,"log_dups_size":0,"ondisk_log_size":7059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'9764","reported_seq":17659,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:51.024710+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:54:51.024710+0000","last_peered":"2023-09-23T03:54:51.024710+0000","last_clean":"2023-09-23T03:54:51.024710+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:54:51.024710+0000","last_undegraded":"2023-09-23T03:54:51.024710+0000","last_fullsized":"2023-09-23T03:54:51.024710+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9764,"log_dups_size":0,"ondisk_log_size":9764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'260646","reported_seq":518202,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:56.770917+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:54:56.770917+0000","last_peered":"2023-09-23T03:54:56.770917+0000","last_clean":"2023-09-23T03:54:56.770917+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:54:56.770917+0000","last_undegraded":"2023-09-23T03:54:56.770917+0000","last_fullsized":"2023-09-23T03:54:56.770917+0000","mapping_epoch":18,"log_start":"155'250624","ondisk_log_start":"155'250624","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5143982080,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":260048,"num_read_kb":1019096,"num_write":257622,"num_write_kb":1030488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2260,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.912080+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:28.912080+0000","last_peered":"2023-09-23T03:53:28.912080+0000","last_clean":"2023-09-23T03:53:28.912080+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:28.912080+0000","last_undegraded":"2023-09-23T03:53:28.912080+0000","last_fullsized":"2023-09-23T03:53:28.912080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'272198","reported_seq":541163,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:53.418528+0000","last_change":"2023-09-23T03:54:44.851059+0000","last_active":"2023-09-23T03:54:53.418528+0000","last_peered":"2023-09-23T03:54:53.418528+0000","last_clean":"2023-09-23T03:54:53.418528+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:54:53.418528+0000","last_undegraded":"2023-09-23T03:54:53.418528+0000","last_fullsized":"2023-09-23T03:54:53.418528+0000","mapping_epoch":18,"log_start":"155'262126","ondisk_log_start":"155'262126","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'271729","last_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_deep_scrub":"158'271729","last_deep_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_clean_scrub_stamp":"2023-09-23T03:54:44.850811+0000","objects_scrubbed":4224,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":498,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:05:20.739642+0000","scrub_duration":497.573879159,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5377794048,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":271545,"num_read_kb":1063796,"num_write":269017,"num_write_kb":1076068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5812,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:55.597732+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:54:55.597732+0000","last_peered":"2023-09-23T03:54:55.597732+0000","last_clean":"2023-09-23T03:54:55.597732+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:54:55.597732+0000","last_undegraded":"2023-09-23T03:54:55.597732+0000","last_fullsized":"2023-09-23T03:54:55.597732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'266258","reported_seq":529386,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:56.743509+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:54:56.743509+0000","last_peered":"2023-09-23T03:54:56.743509+0000","last_clean":"2023-09-23T03:54:56.743509+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:54:56.743509+0000","last_undegraded":"2023-09-23T03:54:56.743509+0000","last_fullsized":"2023-09-23T03:54:56.743509+0000","mapping_epoch":18,"log_start":"155'256212","ondisk_log_start":"155'256212","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5244350464,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":265639,"num_read_kb":1041744,"num_write":263174,"num_write_kb":1052696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":4996,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:55.556513+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:54:55.556513+0000","last_peered":"2023-09-23T03:54:55.556513+0000","last_clean":"2023-09-23T03:54:55.556513+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:54:55.556513+0000","last_undegraded":"2023-09-23T03:54:55.556513+0000","last_fullsized":"2023-09-23T03:54:55.556513+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'246963","reported_seq":490806,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:54:53.428194+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:54:53.428194+0000","last_peered":"2023-09-23T03:54:53.428194+0000","last_clean":"2023-09-23T03:54:53.428194+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:54:53.428194+0000","last_undegraded":"2023-09-23T03:54:53.428194+0000","last_fullsized":"2023-09-23T03:54:53.428194+0000","mapping_epoch":18,"log_start":"155'236914","ondisk_log_start":"155'236914","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":2883,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 210s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4871077907,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":246294,"num_read_kb":965932,"num_write":244052,"num_write_kb":976202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'10378","reported_seq":18676,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:51.090000+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:54:51.090000+0000","last_peered":"2023-09-23T03:54:51.090000+0000","last_clean":"2023-09-23T03:54:51.090000+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:54:51.090000+0000","last_undegraded":"2023-09-23T03:54:51.090000+0000","last_fullsized":"2023-09-23T03:54:51.090000+0000","mapping_epoch":13,"log_start":"43'300","ondisk_log_start":"43'300","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":330,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:28.072773+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:54:28.072773+0000","last_peered":"2023-09-23T03:54:28.072773+0000","last_clean":"2023-09-23T03:54:28.072773+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:54:28.072773+0000","last_undegraded":"2023-09-23T03:54:28.072773+0000","last_fullsized":"2023-09-23T03:54:28.072773+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20637204499,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1043526,"num_read_kb":4090568,"num_write":1033865,"num_write_kb":4135454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36030332928,"data_stored":98815401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40189,"ondisk_log_size":40189,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45958,"ondisk_log_size":45958,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739273,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12798044,"kb_used_data":11731284,"kb_used_omap":1245,"kb_used_meta":1065506,"kb_avail":81573796,"statfs":{"total":96636764160,"available":83531567104,"internally_reserved":0,"allocated":12012834816,"data_stored":32939155406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1091078684},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.86899999999999999,"15min":0.96599999999999997},"min":{"1min":0.154,"5min":0.154,"15min":0.154},"max":{"1min":4.21,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.60599999999999998,"15min":0.60299999999999998},"min":{"1min":0.25,"5min":0.19900000000000001,"15min":0.19500000000000001},"max":{"1min":4.1150000000000002,"5min":13.618,"15min":13.618},"last":0.20899999999999999}]},{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.57999999999999996,"15min":0.60099999999999998},"min":{"1min":0.24399999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":1.3779999999999999,"5min":10.528,"15min":10.528},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.85499999999999998,"15min":0.83099999999999996},"min":{"1min":0.30299999999999999,"5min":0.247,"15min":0.20999999999999999},"max":{"1min":1.413,"5min":10.625,"15min":10.625},"last":0.42599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739270,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12812728,"kb_used_data":11729712,"kb_used_omap":1256,"kb_used_meta":1081751,"kb_avail":81559112,"statfs":{"total":96636764160,"available":83516530688,"internally_reserved":0,"allocated":12011225088,"data_stored":32939726080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1107713218},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":33.228999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.8129999999999999,"5min":0.90300000000000002,"15min":0.80400000000000005},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":0.85799999999999998,"15min":0.76700000000000002},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":33.505000000000003,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.316},{"interface":"front","average":{"1min":1.839,"5min":0.89200000000000002,"15min":0.79700000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":33.408999999999999,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359739269,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13034348,"kb_used_data":11731556,"kb_used_omap":1251,"kb_used_meta":1301532,"kb_avail":81337492,"statfs":{"total":96636764160,"available":83289591808,"internally_reserved":0,"allocated":12013113344,"data_stored":32939753521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1332768961},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,9,9,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.64400000000000002,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.17799999999999999},"max":{"1min":0.56100000000000005,"5min":7.0880000000000001,"15min":11.715},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.27000000000000002,"5min":0.874,"15min":0.89600000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":7.0469999999999997,"15min":12.058},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:54:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.66700000000000004,"15min":0.56399999999999995},"min":{"1min":0.219,"5min":0.219,"15min":0.20100000000000001},"max":{"1min":0.54300000000000004,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.53200000000000003,"15min":0.55100000000000005},"min":{"1min":0.25,"5min":0.24399999999999999,"15min":0.22600000000000001},"max":{"1min":0.57699999999999996,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12010737664,"data_stored":32938496000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12008255488,"data_stored":32938422272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12011339776,"data_stored":32938483712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:54:59.185 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:54:59.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:54:59.300 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.292+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:54:59.323 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.316+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:54:59.323 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.316+0000 7fa394779640 -1 reset not still connected to 0x55a11e5025b0 2023-09-23T03:54:59.345 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11e73f520 2023-09-23T03:54:59.345 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:54:59.345 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11ed60d00 2023-09-23T03:54:59.345 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:54:59.345 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11f47bc70 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11f747e10 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a11fe95ee0 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:54:59.346 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:54:59.347 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:54:59.340+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:55:15.923 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:55:15.920+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:55:15.928 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:55:15.924+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:55:19.187 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:55:19.976 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:55:19.977 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:55:20.018 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2424,"stamp":"2023-09-23T03:55:18.313415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20655208980,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1201035,"num_read_kb":4236414,"num_write":1072368,"num_write_kb":4187145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86314,"ondisk_log_size":86314,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":38908856,"kb_used_data":35235616,"kb_used_omap":3754,"kb_used_meta":3669461,"kb_avail":244206664,"statfs":{"total":289910292480,"available":250067623936,"internally_reserved":0,"allocated":36081270784,"data_stored":98819314943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":3757528991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,8,11,15,10,17,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10993664,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2803,"num_read_kb":11164,"num_write":2638,"num_write_kb":10552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002363"},"pg_stats":[{"pgid":"2.7","version":"158'8108","reported_seq":13204,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:12.047783+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:55:12.047783+0000","last_peered":"2023-09-23T03:55:12.047783+0000","last_clean":"2023-09-23T03:55:12.047783+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:55:12.047783+0000","last_undegraded":"2023-09-23T03:55:12.047783+0000","last_fullsized":"2023-09-23T03:55:12.047783+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'5409","reported_seq":9332,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:10.916401+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:55:10.916401+0000","last_peered":"2023-09-23T03:55:10.916401+0000","last_clean":"2023-09-23T03:55:10.916401+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:55:10.916401+0000","last_undegraded":"2023-09-23T03:55:10.916401+0000","last_fullsized":"2023-09-23T03:55:10.916401+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'7083","reported_seq":12131,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:12.044650+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:55:12.044650+0000","last_peered":"2023-09-23T03:55:12.044650+0000","last_clean":"2023-09-23T03:55:12.044650+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:55:12.044650+0000","last_undegraded":"2023-09-23T03:55:12.044650+0000","last_fullsized":"2023-09-23T03:55:12.044650+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'9796","reported_seq":17707,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:12.059681+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:55:12.059681+0000","last_peered":"2023-09-23T03:55:12.059681+0000","last_clean":"2023-09-23T03:55:12.059681+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:55:12.059681+0000","last_undegraded":"2023-09-23T03:55:12.059681+0000","last_fullsized":"2023-09-23T03:55:12.059681+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9796,"log_dups_size":0,"ondisk_log_size":9796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'261683","reported_seq":520316,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:17.020970+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:55:17.020970+0000","last_peered":"2023-09-23T03:55:17.020970+0000","last_clean":"2023-09-23T03:55:17.020970+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:55:17.020970+0000","last_undegraded":"2023-09-23T03:55:17.020970+0000","last_fullsized":"2023-09-23T03:55:17.020970+0000","mapping_epoch":18,"log_start":"155'251635","ondisk_log_start":"155'251635","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5148221440,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":261120,"num_read_kb":1023368,"num_write":258659,"num_write_kb":1034636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2260,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:53:28.912080+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:53:28.912080+0000","last_peered":"2023-09-23T03:53:28.912080+0000","last_clean":"2023-09-23T03:53:28.912080+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:53:28.912080+0000","last_undegraded":"2023-09-23T03:53:28.912080+0000","last_fullsized":"2023-09-23T03:53:28.912080+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'273289","reported_seq":543401,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:13.726248+0000","last_change":"2023-09-23T03:54:44.851059+0000","last_active":"2023-09-23T03:55:13.726248+0000","last_peered":"2023-09-23T03:55:13.726248+0000","last_clean":"2023-09-23T03:55:13.726248+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:55:13.726248+0000","last_undegraded":"2023-09-23T03:55:13.726248+0000","last_fullsized":"2023-09-23T03:55:13.726248+0000","mapping_epoch":18,"log_start":"155'263239","ondisk_log_start":"155'263239","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'271729","last_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_deep_scrub":"158'271729","last_deep_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_clean_scrub_stamp":"2023-09-23T03:54:44.850811+0000","objects_scrubbed":4224,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":498,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:05:20.739642+0000","scrub_duration":497.573879159,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5382369280,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":272700,"num_read_kb":1068408,"num_write":270108,"num_write_kb":1080432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5820,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:16.036377+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:55:16.036377+0000","last_peered":"2023-09-23T03:55:16.036377+0000","last_clean":"2023-09-23T03:55:16.036377+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:55:16.036377+0000","last_undegraded":"2023-09-23T03:55:16.036377+0000","last_fullsized":"2023-09-23T03:55:16.036377+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'267317","reported_seq":531581,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:17.020825+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:55:17.020825+0000","last_peered":"2023-09-23T03:55:17.020825+0000","last_clean":"2023-09-23T03:55:17.020825+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:55:17.020825+0000","last_undegraded":"2023-09-23T03:55:17.020825+0000","last_fullsized":"2023-09-23T03:55:17.020825+0000","mapping_epoch":18,"log_start":"155'257230","ondisk_log_start":"155'257230","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5248700416,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":266771,"num_read_kb":1046252,"num_write":264233,"num_write_kb":1056932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":5004,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:16.058946+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:55:16.058946+0000","last_peered":"2023-09-23T03:55:16.058946+0000","last_clean":"2023-09-23T03:55:16.058946+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:55:16.058946+0000","last_undegraded":"2023-09-23T03:55:16.058946+0000","last_fullsized":"2023-09-23T03:55:16.058946+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'247973","reported_seq":492869,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:55:13.713474+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:55:13.713474+0000","last_peered":"2023-09-23T03:55:13.713474+0000","last_clean":"2023-09-23T03:55:13.713474+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:55:13.713474+0000","last_undegraded":"2023-09-23T03:55:13.713474+0000","last_fullsized":"2023-09-23T03:55:13.713474+0000","mapping_epoch":18,"log_start":"155'237927","ondisk_log_start":"155'237927","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3149,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4875313171,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":247329,"num_read_kb":970036,"num_write":245062,"num_write_kb":980242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'10416","reported_seq":18734,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:12.101821+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:55:12.101821+0000","last_peered":"2023-09-23T03:55:12.101821+0000","last_clean":"2023-09-23T03:55:12.101821+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:55:12.101821+0000","last_undegraded":"2023-09-23T03:55:12.101821+0000","last_fullsized":"2023-09-23T03:55:12.101821+0000","mapping_epoch":13,"log_start":"43'400","ondisk_log_start":"43'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":330,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:28.072773+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:54:28.072773+0000","last_peered":"2023-09-23T03:54:28.072773+0000","last_clean":"2023-09-23T03:54:28.072773+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:54:28.072773+0000","last_undegraded":"2023-09-23T03:54:28.072773+0000","last_fullsized":"2023-09-23T03:54:28.072773+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20654604307,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1047920,"num_read_kb":4108064,"num_write":1038062,"num_write_kb":4152242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36074635264,"data_stored":98816069632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":45984,"ondisk_log_size":45984,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739277,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12884596,"kb_used_data":11745644,"kb_used_omap":1245,"kb_used_meta":1137698,"kb_avail":81487244,"statfs":{"total":96636764160,"available":83442937856,"internally_reserved":0,"allocated":12027539456,"data_stored":32939384782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1165003292},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.999,"15min":0.94999999999999996},"min":{"1min":0.161,"5min":0.154,"15min":0.154},"max":{"1min":6.2279999999999998,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.308},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.68200000000000005,"15min":0.59499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1500000000000004,"5min":13.618,"15min":13.618},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.60399999999999998,"15min":0.58299999999999996},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5110000000000001,"5min":10.528,"15min":10.528},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.92400000000000004,"15min":0.84699999999999998},"min":{"1min":0.30099999999999999,"5min":0.247,"15min":0.223},"max":{"1min":6.0190000000000001,"5min":10.625,"15min":10.625},"last":0.28599999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739274,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12921392,"kb_used_data":11744040,"kb_used_omap":1256,"kb_used_meta":1176087,"kb_avail":81450448,"statfs":{"total":96636764160,"available":83405258752,"internally_reserved":0,"allocated":12025896960,"data_stored":32939967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1204313282},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,11,1,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.9259999999999999,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.184,"5min":0.17999999999999999,"15min":0.161},"max":{"1min":33.228999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.8129999999999999,"5min":0.90300000000000002,"15min":0.80400000000000005},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Sep 23 03:54:10 2023","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":0.85799999999999998,"15min":0.76700000000000002},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":33.505000000000003,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.839,"5min":0.89200000000000002,"15min":0.79700000000000004},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.17499999999999999},"max":{"1min":33.408999999999999,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.373}]}]},{"osd":0,"up_from":8,"seq":34359739273,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13102868,"kb_used_data":11745932,"kb_used_omap":1251,"kb_used_meta":1355676,"kb_avail":81268972,"statfs":{"total":96636764160,"available":83219427328,"internally_reserved":0,"allocated":12027834368,"data_stored":32939962417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1388212417},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,11,4,9,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":3.1899999999999999,"5min":7.0880000000000001,"15min":11.715},"last":0.307},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.84799999999999998,"15min":0.85099999999999998},"min":{"1min":0.223,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.3399999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.58199999999999996},"min":{"1min":0.254,"5min":0.219,"15min":0.216},"max":{"1min":2.766,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":6.2279999999999998,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12025516032,"data_stored":32938659840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12023148544,"data_stored":32938688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12025970688,"data_stored":32938721280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:55:20.020 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:55:20.020 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:55:40.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:55:40.625 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:55:40.626 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:55:40.652 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2435,"stamp":"2023-09-23T03:55:40.317677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20674722324,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1205876,"num_read_kb":4255682,"num_write":1077079,"num_write_kb":4205989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86490,"ondisk_log_size":86490,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39198644,"kb_used_data":35282008,"kb_used_omap":3754,"kb_used_meta":3912853,"kb_avail":243916876,"statfs":{"total":289910292480,"available":249770881024,"internally_reserved":0,"allocated":36128776192,"data_stored":98820056319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4006762399},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,7,2,3,9,27,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10821632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2665,"num_read_kb":10596,"num_write":2604,"num_write_kb":10416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002403"},"pg_stats":[{"pgid":"2.7","version":"158'8132","reported_seq":13243,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.521015+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:55:36.521015+0000","last_peered":"2023-09-23T03:55:36.521015+0000","last_clean":"2023-09-23T03:55:36.521015+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:55:36.521015+0000","last_undegraded":"2023-09-23T03:55:36.521015+0000","last_fullsized":"2023-09-23T03:55:36.521015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'5417","reported_seq":9345,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:35.252704+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:55:35.252704+0000","last_peered":"2023-09-23T03:55:35.252704+0000","last_clean":"2023-09-23T03:55:35.252704+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:55:35.252704+0000","last_undegraded":"2023-09-23T03:55:35.252704+0000","last_fullsized":"2023-09-23T03:55:35.252704+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'7108","reported_seq":12172,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.520901+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:55:36.520901+0000","last_peered":"2023-09-23T03:55:36.520901+0000","last_clean":"2023-09-23T03:55:36.520901+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:55:36.520901+0000","last_undegraded":"2023-09-23T03:55:36.520901+0000","last_fullsized":"2023-09-23T03:55:36.520901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7108,"log_dups_size":0,"ondisk_log_size":7108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'9829","reported_seq":17761,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.521704+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:55:36.521704+0000","last_peered":"2023-09-23T03:55:36.521704+0000","last_clean":"2023-09-23T03:55:36.521704+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:55:36.521704+0000","last_undegraded":"2023-09-23T03:55:36.521704+0000","last_fullsized":"2023-09-23T03:55:36.521704+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9829,"log_dups_size":0,"ondisk_log_size":9829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"158'262639","reported_seq":522325,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:37.238832+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:55:37.238832+0000","last_peered":"2023-09-23T03:55:37.238832+0000","last_clean":"2023-09-23T03:55:37.238832+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:55:37.238832+0000","last_undegraded":"2023-09-23T03:55:37.238832+0000","last_fullsized":"2023-09-23T03:55:37.238832+0000","mapping_epoch":18,"log_start":"155'252545","ondisk_log_start":"155'252545","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5152202752,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":262174,"num_read_kb":1027568,"num_write":259615,"num_write_kb":1038460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2261,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:38.937332+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:54:38.937332+0000","last_peered":"2023-09-23T03:54:38.937332+0000","last_clean":"2023-09-23T03:54:38.937332+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:54:38.937332+0000","last_undegraded":"2023-09-23T03:54:38.937332+0000","last_fullsized":"2023-09-23T03:54:38.937332+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"158'274661","reported_seq":546186,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:38.972152+0000","last_change":"2023-09-23T03:54:44.851059+0000","last_active":"2023-09-23T03:55:38.972152+0000","last_peered":"2023-09-23T03:55:38.972152+0000","last_clean":"2023-09-23T03:55:38.972152+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:55:38.972152+0000","last_undegraded":"2023-09-23T03:55:38.972152+0000","last_fullsized":"2023-09-23T03:55:38.972152+0000","mapping_epoch":18,"log_start":"155'264658","ondisk_log_start":"155'264658","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'271729","last_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_deep_scrub":"158'271729","last_deep_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_clean_scrub_stamp":"2023-09-23T03:54:44.850811+0000","objects_scrubbed":4224,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":498,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:05:20.739642+0000","scrub_duration":497.573879159,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5388058624,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":274112,"num_read_kb":1074028,"num_write":271480,"num_write_kb":1085920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5828,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.520518+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:55:36.520518+0000","last_peered":"2023-09-23T03:55:36.520518+0000","last_clean":"2023-09-23T03:55:36.520518+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:55:36.520518+0000","last_undegraded":"2023-09-23T03:55:36.520518+0000","last_fullsized":"2023-09-23T03:55:36.520518+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"158'268437","reported_seq":533722,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:37.222895+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:55:37.222895+0000","last_peered":"2023-09-23T03:55:37.222895+0000","last_clean":"2023-09-23T03:55:37.222895+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:55:37.222895+0000","last_undegraded":"2023-09-23T03:55:37.222895+0000","last_fullsized":"2023-09-23T03:55:37.222895+0000","mapping_epoch":18,"log_start":"155'258354","ondisk_log_start":"155'258354","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5253316608,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":267793,"num_read_kb":1050316,"num_write":265353,"num_write_kb":1061412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":5012,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.519991+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:55:36.519991+0000","last_peered":"2023-09-23T03:55:36.519991+0000","last_clean":"2023-09-23T03:55:36.519991+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:55:36.519991+0000","last_undegraded":"2023-09-23T03:55:36.519991+0000","last_fullsized":"2023-09-23T03:55:36.519991+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"158'249236","reported_seq":495506,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:55:39.041845+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:55:39.041845+0000","last_peered":"2023-09-23T03:55:39.041845+0000","last_clean":"2023-09-23T03:55:39.041845+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:55:39.041845+0000","last_undegraded":"2023-09-23T03:55:39.041845+0000","last_fullsized":"2023-09-23T03:55:39.041845+0000","mapping_epoch":18,"log_start":"155'239139","ondisk_log_start":"155'239139","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3387,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 255s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4880539667,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":248682,"num_read_kb":975420,"num_write":246325,"num_write_kb":985294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"158'10456","reported_seq":18800,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:36.521173+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:55:36.521173+0000","last_peered":"2023-09-23T03:55:36.521173+0000","last_clean":"2023-09-23T03:55:36.521173+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:55:36.521173+0000","last_undegraded":"2023-09-23T03:55:36.521173+0000","last_fullsized":"2023-09-23T03:55:36.521173+0000","mapping_epoch":13,"log_start":"43'400","ondisk_log_start":"43'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":330,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-09-23T03:54:28.072773+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:54:28.072773+0000","last_peered":"2023-09-23T03:54:28.072773+0000","last_clean":"2023-09-23T03:54:28.072773+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:54:28.072773+0000","last_undegraded":"2023-09-23T03:54:28.072773+0000","last_fullsized":"2023-09-23T03:54:28.072773+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20674117651,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1052761,"num_read_kb":4127332,"num_write":1042773,"num_write_kb":4171086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36122132480,"data_stored":98816815104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40277,"ondisk_log_size":40277,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":46114,"ondisk_log_size":46114,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739281,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12975184,"kb_used_data":11759880,"kb_used_omap":1245,"kb_used_meta":1214050,"kb_avail":81396656,"statfs":{"total":96636764160,"available":83350175744,"internally_reserved":0,"allocated":12042117120,"data_stored":32939618254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1243187740},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.999,"15min":0.94999999999999996},"min":{"1min":0.161,"5min":0.154,"15min":0.154},"max":{"1min":6.2279999999999998,"5min":14.617000000000001,"15min":14.617000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.68200000000000005,"15min":0.59499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1500000000000004,"5min":13.618,"15min":13.618},"last":0.26500000000000001}]},{"osd":1,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.60399999999999998,"15min":0.58299999999999996},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5110000000000001,"5min":10.528,"15min":10.528},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.92400000000000004,"15min":0.84699999999999998},"min":{"1min":0.30099999999999999,"5min":0.247,"15min":0.223},"max":{"1min":6.0190000000000001,"5min":10.625,"15min":10.625},"last":0.41899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739279,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13033680,"kb_used_data":11761928,"kb_used_omap":1256,"kb_used_meta":1270487,"kb_avail":81338160,"statfs":{"total":96636764160,"available":83290275840,"internally_reserved":0,"allocated":12044214272,"data_stored":32940201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1300978882},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,1,7,12,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.876,"15min":0.88},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.161},"max":{"1min":2.8799999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.84299999999999997,"15min":0.753},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":2.8439999999999999,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Sep 23 03:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.81399999999999995,"15min":0.749},"min":{"1min":0.27800000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.79,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.80200000000000005,"15min":0.79000000000000004},"min":{"1min":0.221,"5min":0.221,"15min":0.17499999999999999},"max":{"1min":0.55700000000000005,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.39600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739277,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13189780,"kb_used_data":11760200,"kb_used_omap":1251,"kb_used_meta":1428316,"kb_avail":81182060,"statfs":{"total":96636764160,"available":83130429440,"internally_reserved":0,"allocated":12042444800,"data_stored":32940236849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1462595777},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,0,2,2,15],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":3.1899999999999999,"5min":7.0880000000000001,"15min":11.715},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.84799999999999998,"15min":0.85099999999999998},"min":{"1min":0.223,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.3399999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.32000000000000001}]},{"osd":2,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.58199999999999996},"min":{"1min":0.254,"5min":0.219,"15min":0.216},"max":{"1min":2.766,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":6.2279999999999998,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.37}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12040138752,"data_stored":32938917888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12041531392,"data_stored":32938958848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12040462336,"data_stored":32938938368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:55:40.654 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:55:40.654 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:56:00.655 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:56:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:56:01.356 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:56:01.395 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2446,"stamp":"2023-09-23T03:56:00.325738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20691872276,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1210149,"num_read_kb":4272682,"num_write":1081245,"num_write_kb":4222653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86486,"ondisk_log_size":86486,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39479988,"kb_used_data":35324184,"kb_used_omap":3754,"kb_used_meta":4152021,"kb_avail":243635532,"statfs":{"total":289910292480,"available":249482784768,"internally_reserved":0,"allocated":36171964416,"data_stored":98820439914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4251670431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,13,10,11,17,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8609792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2116,"num_read_kb":8412,"num_write":2084,"num_write_kb":8336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001724"},"pg_stats":[{"pgid":"2.7","version":"159'8147","reported_seq":13271,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:56.999994+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:55:56.999994+0000","last_peered":"2023-09-23T03:55:56.999994+0000","last_clean":"2023-09-23T03:55:56.999994+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:55:56.999994+0000","last_undegraded":"2023-09-23T03:55:56.999994+0000","last_fullsized":"2023-09-23T03:55:56.999994+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8147,"log_dups_size":0,"ondisk_log_size":8147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'5422","reported_seq":9355,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:54.358624+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:55:54.358624+0000","last_peered":"2023-09-23T03:55:54.358624+0000","last_clean":"2023-09-23T03:55:54.358624+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:55:54.358624+0000","last_undegraded":"2023-09-23T03:55:54.358624+0000","last_fullsized":"2023-09-23T03:55:54.358624+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'7123","reported_seq":12200,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:56.996244+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:55:56.996244+0000","last_peered":"2023-09-23T03:55:56.996244+0000","last_clean":"2023-09-23T03:55:56.996244+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:55:56.996244+0000","last_undegraded":"2023-09-23T03:55:56.996244+0000","last_fullsized":"2023-09-23T03:55:56.996244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7123,"log_dups_size":0,"ondisk_log_size":7123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'9849","reported_seq":17798,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:56.996634+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:55:56.996634+0000","last_peered":"2023-09-23T03:55:56.996634+0000","last_clean":"2023-09-23T03:55:56.996634+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:55:56.996634+0000","last_undegraded":"2023-09-23T03:55:56.996634+0000","last_fullsized":"2023-09-23T03:55:56.996634+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9849,"log_dups_size":0,"ondisk_log_size":9849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"159'263669","reported_seq":524489,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:57.364381+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:55:57.364381+0000","last_peered":"2023-09-23T03:55:57.364381+0000","last_clean":"2023-09-23T03:55:57.364381+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:55:57.364381+0000","last_undegraded":"2023-09-23T03:55:57.364381+0000","last_fullsized":"2023-09-23T03:55:57.364381+0000","mapping_epoch":18,"log_start":"156'253660","ondisk_log_start":"156'253660","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5156429824,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":263307,"num_read_kb":1032080,"num_write":260645,"num_write_kb":1042580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2262,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:22.064493+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:55:22.064493+0000","last_peered":"2023-09-23T03:55:22.064493+0000","last_clean":"2023-09-23T03:55:22.064493+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:55:22.064493+0000","last_undegraded":"2023-09-23T03:55:22.064493+0000","last_fullsized":"2023-09-23T03:55:22.064493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"159'275778","reported_seq":548389,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:59.239848+0000","last_change":"2023-09-23T03:54:44.851059+0000","last_active":"2023-09-23T03:55:59.239848+0000","last_peered":"2023-09-23T03:55:59.239848+0000","last_clean":"2023-09-23T03:55:59.239848+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:55:59.239848+0000","last_undegraded":"2023-09-23T03:55:59.239848+0000","last_fullsized":"2023-09-23T03:55:59.239848+0000","mapping_epoch":18,"log_start":"156'265672","ondisk_log_start":"156'265672","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'271729","last_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_deep_scrub":"158'271729","last_deep_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_clean_scrub_stamp":"2023-09-23T03:54:44.850811+0000","objects_scrubbed":4224,"log_size":10106,"log_dups_size":0,"ondisk_log_size":10106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":498,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:05:20.739642+0000","scrub_duration":497.573879159,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5392678912,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":275198,"num_read_kb":1078352,"num_write":272597,"num_write_kb":1090388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5837,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:56.998597+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:55:56.998597+0000","last_peered":"2023-09-23T03:55:56.998597+0000","last_clean":"2023-09-23T03:55:56.998597+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:55:56.998597+0000","last_undegraded":"2023-09-23T03:55:56.998597+0000","last_fullsized":"2023-09-23T03:55:56.998597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"159'269501","reported_seq":535889,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:57.370242+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:55:57.370242+0000","last_peered":"2023-09-23T03:55:57.370242+0000","last_clean":"2023-09-23T03:55:57.370242+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:55:57.370242+0000","last_undegraded":"2023-09-23T03:55:57.370242+0000","last_fullsized":"2023-09-23T03:55:57.370242+0000","mapping_epoch":18,"log_start":"156'259468","ondisk_log_start":"156'259468","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5257670656,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":268894,"num_read_kb":1054684,"num_write":266417,"num_write_kb":1065668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":5021,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:56.999134+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:55:56.999134+0000","last_peered":"2023-09-23T03:55:56.999134+0000","last_clean":"2023-09-23T03:55:56.999134+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:55:56.999134+0000","last_undegraded":"2023-09-23T03:55:56.999134+0000","last_fullsized":"2023-09-23T03:55:56.999134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"159'250191","reported_seq":497433,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-09-23T03:55:59.234950+0000","last_change":"2023-09-23T03:51:22.988628+0000","last_active":"2023-09-23T03:55:59.234950+0000","last_peered":"2023-09-23T03:55:59.234950+0000","last_clean":"2023-09-23T03:55:59.234950+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:55:59.234950+0000","last_undegraded":"2023-09-23T03:55:59.234950+0000","last_fullsized":"2023-09-23T03:55:59.234950+0000","mapping_epoch":18,"log_start":"155'240146","ondisk_log_start":"155'240146","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-23T02:39:56.262767+0000","last_clean_scrub_stamp":"2023-09-23T02:39:56.262767+0000","objects_scrubbed":3713,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 276s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4884488211,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":249635,"num_read_kb":979216,"num_write":247280,"num_write_kb":989114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'10481","reported_seq":18846,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:55:57.003867+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:55:57.003867+0000","last_peered":"2023-09-23T03:55:57.003867+0000","last_clean":"2023-09-23T03:55:57.003867+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:55:57.003867+0000","last_undegraded":"2023-09-23T03:55:57.003867+0000","last_fullsized":"2023-09-23T03:55:57.003867+0000","mapping_epoch":13,"log_start":"43'400","ondisk_log_start":"43'400","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":331,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:38.961998+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:54:38.961998+0000","last_peered":"2023-09-23T03:54:38.961998+0000","last_clean":"2023-09-23T03:54:38.961998+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:54:38.961998+0000","last_undegraded":"2023-09-23T03:54:38.961998+0000","last_fullsized":"2023-09-23T03:54:38.961998+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20691267603,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1057034,"num_read_kb":4144332,"num_write":1046939,"num_write_kb":4187750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36165648384,"data_stored":98817208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40193,"ondisk_log_size":40193,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":46194,"ondisk_log_size":46194,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739285,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13061552,"kb_used_data":11774056,"kb_used_omap":1245,"kb_used_meta":1286242,"kb_avail":81310288,"statfs":{"total":96636764160,"available":83261734912,"internally_reserved":0,"allocated":12056633344,"data_stored":32939754311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1317112348},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.999,"15min":0.94999999999999996},"min":{"1min":0.161,"5min":0.154,"15min":0.154},"max":{"1min":6.2279999999999998,"5min":14.617000000000001,"15min":14.617000000000001},"last":4.3369999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.68200000000000005,"15min":0.59499999999999997},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.19500000000000001},"max":{"1min":6.1500000000000004,"5min":13.618,"15min":13.618},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Sep 23 03:55:08 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.60399999999999998,"15min":0.58299999999999996},"min":{"1min":0.28499999999999998,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.5110000000000001,"5min":10.528,"15min":10.528},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.92400000000000004,"15min":0.84699999999999998},"min":{"1min":0.30099999999999999,"5min":0.247,"15min":0.223},"max":{"1min":6.0190000000000001,"5min":10.625,"15min":10.625},"last":0.57699999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739283,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13137984,"kb_used_data":11775992,"kb_used_omap":1256,"kb_used_meta":1360727,"kb_avail":81233856,"statfs":{"total":96636764160,"available":83183468544,"internally_reserved":0,"allocated":12058615808,"data_stored":32940345465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1393384642},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,10,2,5,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.876,"15min":0.88},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.161},"max":{"1min":2.8799999999999999,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.84299999999999997,"15min":0.753},"min":{"1min":0.23300000000000001,"5min":0.20899999999999999,"15min":0.185},"max":{"1min":2.8439999999999999,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.81399999999999995,"15min":0.749},"min":{"1min":0.27800000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":2.79,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.80200000000000005,"15min":0.79000000000000004},"min":{"1min":0.221,"5min":0.221,"15min":0.17499999999999999},"max":{"1min":0.55700000000000005,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739281,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13280452,"kb_used_data":11774136,"kb_used_omap":1251,"kb_used_meta":1505052,"kb_avail":81091388,"statfs":{"total":96636764160,"available":83037581312,"internally_reserved":0,"allocated":12056715264,"data_stored":32940340138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1541173441},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,2,4,8,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.72199999999999998,"15min":0.71199999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":3.1899999999999999,"5min":7.0880000000000001,"15min":11.715},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.84799999999999998,"15min":0.85099999999999998},"min":{"1min":0.223,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.3399999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.24299999999999999}]},{"osd":2,"last update":"Sat Sep 23 03:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69699999999999995,"15min":0.58199999999999996},"min":{"1min":0.254,"5min":0.219,"15min":0.216},"max":{"1min":2.766,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.624,"15min":0.58599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":6.2279999999999998,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12054601728,"data_stored":32939036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12056064000,"data_stored":32939102208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12054982656,"data_stored":32939069440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:56:01.397 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-09-23T02:39:56.262767+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=2, tm_min=39, tm_sec=56, tm_wday=5, tm_yday=266, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=3, tm_wday=5, tm_yday=266, tm_isdst=0) 2023-09-23T03:56:01.397 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-23T03:56:13.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a11e7d3ee0 2023-09-23T03:56:13.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a11ed61ad0 2023-09-23T03:56:13.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c1a0 2023-09-23T03:56:13.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:56:13.096 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a1206c5a00 2023-09-23T03:56:13.097 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:56:13.097 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a121ee5ee0 2023-09-23T03:56:13.097 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:13.094+0000 7fa394779640 -1 reset not still connected to 0x55a121ef1930 2023-09-23T03:56:14.626 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:14.626+0000 7fa394779640 -1 reset not still connected to 0x55a11ddfd450 2023-09-23T03:56:14.646 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:14.638+0000 7fa394779640 -1 reset not still connected to 0x55a11e4829c0 2023-09-23T03:56:14.647 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:14.638+0000 7fa394779640 -1 reset not still connected to 0x55a11f74c8f0 2023-09-23T03:56:14.647 INFO:tasks.ceph.osd.1.smithi002.stderr:2023-09-23T03:56:14.638+0000 7fa394779640 -1 reset not still connected to 0x55a1217f5c70 2023-09-23T03:56:21.398 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-23T03:56:22.044 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-09-23T03:56:22.046 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-09-23T03:56:22.087 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":2456,"stamp":"2023-09-23T03:56:20.338777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20709566996,"num_objects":16428,"num_object_clones":12200,"num_object_copies":49057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16428,"num_whiteouts":0,"num_read":1214369,"num_read_kb":4289442,"num_write":1085543,"num_write_kb":4239845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86652,"ondisk_log_size":86652,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":39727184,"kb_used_data":35368696,"kb_used_omap":3754,"kb_used_meta":4354709,"kb_avail":243388336,"statfs":{"total":289910292480,"available":249229656064,"internally_reserved":0,"allocated":36217544704,"data_stored":98820886378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3844193,"internal_metadata":4459222943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,11,6,15,13,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11243520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2689,"num_read_kb":10656,"num_write":2716,"num_write_kb":10864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007123"},"pg_stats":[{"pgid":"2.7","version":"159'8172","reported_seq":13309,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.476804+0000","last_change":"2023-09-23T03:46:22.556747+0000","last_active":"2023-09-23T03:56:17.476804+0000","last_peered":"2023-09-23T03:56:17.476804+0000","last_clean":"2023-09-23T03:56:17.476804+0000","last_became_active":"2023-09-23T02:39:49.245282+0000","last_became_peered":"2023-09-23T02:39:49.245282+0000","last_unstale":"2023-09-23T03:56:17.476804+0000","last_undegraded":"2023-09-23T03:56:17.476804+0000","last_fullsized":"2023-09-23T03:56:17.476804+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'7614","last_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_deep_scrub":"150'7614","last_deep_scrub_stamp":"2023-09-23T03:46:22.556399+0000","last_clean_scrub_stamp":"2023-09-23T03:46:22.556399+0000","objects_scrubbed":32,"log_size":8172,"log_dups_size":0,"ondisk_log_size":8172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T05:18:44.031635+0000","scrub_duration":0.34745246699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2013,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":23175,"num_read_kb":19307,"num_write":5424,"num_write_kb":5278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2856,"num_omap_keys":39,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'5431","reported_seq":9368,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:16.527739+0000","last_change":"2023-09-23T03:46:24.526599+0000","last_active":"2023-09-23T03:56:16.527739+0000","last_peered":"2023-09-23T03:56:16.527739+0000","last_clean":"2023-09-23T03:56:16.527739+0000","last_became_active":"2023-09-23T02:39:49.245328+0000","last_became_peered":"2023-09-23T02:39:49.245328+0000","last_unstale":"2023-09-23T03:56:16.527739+0000","last_undegraded":"2023-09-23T03:56:16.527739+0000","last_fullsized":"2023-09-23T03:56:16.527739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'5244","last_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_deep_scrub":"150'5244","last_deep_scrub_stamp":"2023-09-23T03:46:24.526341+0000","last_clean_scrub_stamp":"2023-09-23T03:46:24.526341+0000","objects_scrubbed":31,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:41:22.761492+0000","scrub_duration":0.26446435299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2188,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":26451,"num_read_kb":22608,"num_write":7026,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2145,"num_omap_keys":55,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'7148","reported_seq":12237,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.476818+0000","last_change":"2023-09-23T03:46:10.548731+0000","last_active":"2023-09-23T03:56:17.476818+0000","last_peered":"2023-09-23T03:56:17.476818+0000","last_clean":"2023-09-23T03:56:17.476818+0000","last_became_active":"2023-09-23T02:39:49.245207+0000","last_became_peered":"2023-09-23T02:39:49.245207+0000","last_unstale":"2023-09-23T03:56:17.476818+0000","last_undegraded":"2023-09-23T03:56:17.476818+0000","last_fullsized":"2023-09-23T03:56:17.476818+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'6586","last_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_deep_scrub":"150'6586","last_deep_scrub_stamp":"2023-09-23T03:46:10.548562+0000","last_clean_scrub_stamp":"2023-09-23T03:46:10.548562+0000","objects_scrubbed":28,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:38:55.928960+0000","scrub_duration":0.31494029299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1682,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":17285,"num_read_kb":14410,"num_write":3404,"num_write_kb":3273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3073,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'9882","reported_seq":17847,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.477013+0000","last_change":"2023-09-23T03:46:17.511410+0000","last_active":"2023-09-23T03:56:17.477013+0000","last_peered":"2023-09-23T03:56:17.477013+0000","last_clean":"2023-09-23T03:56:17.477013+0000","last_became_active":"2023-09-23T02:39:49.246930+0000","last_became_peered":"2023-09-23T02:39:49.246930+0000","last_unstale":"2023-09-23T03:56:17.477013+0000","last_undegraded":"2023-09-23T03:56:17.477013+0000","last_fullsized":"2023-09-23T03:56:17.477013+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9136","last_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_deep_scrub":"150'9136","last_deep_scrub_stamp":"2023-09-23T03:46:17.511230+0000","last_clean_scrub_stamp":"2023-09-23T03:46:17.511230+0000","objects_scrubbed":27,"log_size":9882,"log_dups_size":0,"ondisk_log_size":9882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:28:10.346200+0000","scrub_duration":0.28179167399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1319,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":26194,"num_read_kb":21930,"num_write":5148,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4770,"num_omap_keys":94,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"159'264774","reported_seq":526592,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.529290+0000","last_change":"2023-09-23T03:51:18.142396+0000","last_active":"2023-09-23T03:56:17.529290+0000","last_peered":"2023-09-23T03:56:17.529290+0000","last_clean":"2023-09-23T03:56:17.529290+0000","last_became_active":"2023-09-23T02:39:57.544743+0000","last_became_peered":"2023-09-23T02:39:57.544743+0000","last_unstale":"2023-09-23T03:56:17.529290+0000","last_undegraded":"2023-09-23T03:56:17.529290+0000","last_fullsized":"2023-09-23T03:56:17.529290+0000","mapping_epoch":18,"log_start":"156'254678","ondisk_log_start":"156'254678","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'249913","last_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_deep_scrub":"155'249913","last_deep_scrub_stamp":"2023-09-23T03:51:18.142167+0000","last_clean_scrub_stamp":"2023-09-23T03:51:18.142167+0000","objects_scrubbed":4021,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":309,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T06:15:41.760076+0000","scrub_duration":308.16322004699998,"objects_trimmed":0,"snaptrim_duration":0.012032612999999999,"stat_sum":{"num_bytes":5160910848,"num_objects":4021,"num_object_clones":3024,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":264304,"num_read_kb":1036028,"num_write":261750,"num_write_kb":1047000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"150'1928","reported_seq":2262,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:55:22.064493+0000","last_change":"2023-09-23T03:46:09.370163+0000","last_active":"2023-09-23T03:55:22.064493+0000","last_peered":"2023-09-23T03:55:22.064493+0000","last_clean":"2023-09-23T03:55:22.064493+0000","last_became_active":"2023-09-23T02:39:49.243442+0000","last_became_peered":"2023-09-23T02:39:49.243442+0000","last_unstale":"2023-09-23T03:55:22.064493+0000","last_undegraded":"2023-09-23T03:55:22.064493+0000","last_fullsized":"2023-09-23T03:55:22.064493+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1928","last_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_deep_scrub":"150'1928","last_deep_scrub_stamp":"2023-09-23T03:46:09.369985+0000","last_clean_scrub_stamp":"2023-09-23T03:46:09.369985+0000","objects_scrubbed":26,"log_size":1928,"log_dups_size":0,"ondisk_log_size":1928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T03:52:50.627434+0000","scrub_duration":0.396269015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1776,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":11056,"num_read_kb":9230,"num_write":3698,"num_write_kb":3652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1360,"num_omap_keys":20,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"159'276951","reported_seq":550746,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:19.504157+0000","last_change":"2023-09-23T03:54:44.851059+0000","last_active":"2023-09-23T03:56:19.504157+0000","last_peered":"2023-09-23T03:56:19.504157+0000","last_clean":"2023-09-23T03:56:19.504157+0000","last_became_active":"2023-09-23T02:39:57.544772+0000","last_became_peered":"2023-09-23T02:39:57.544772+0000","last_unstale":"2023-09-23T03:56:19.504157+0000","last_undegraded":"2023-09-23T03:56:19.504157+0000","last_fullsized":"2023-09-23T03:56:19.504157+0000","mapping_epoch":18,"log_start":"156'266895","ondisk_log_start":"156'266895","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'271729","last_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_deep_scrub":"158'271729","last_deep_scrub_stamp":"2023-09-23T03:54:44.850811+0000","last_clean_scrub_stamp":"2023-09-23T03:54:44.850811+0000","objects_scrubbed":4224,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":498,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:05:20.739642+0000","scrub_duration":497.573879159,"objects_trimmed":0,"snaptrim_duration":0.0062142719999999999,"stat_sum":{"num_bytes":5397585920,"num_objects":4224,"num_object_clones":3181,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4224,"num_whiteouts":0,"num_read":276377,"num_read_kb":1083040,"num_write":273770,"num_write_kb":1095080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"155'2137","reported_seq":5845,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.477158+0000","last_change":"2023-09-23T03:46:11.630371+0000","last_active":"2023-09-23T03:56:17.477158+0000","last_peered":"2023-09-23T03:56:17.477158+0000","last_clean":"2023-09-23T03:56:17.477158+0000","last_became_active":"2023-09-23T02:39:49.247682+0000","last_became_peered":"2023-09-23T02:39:49.247682+0000","last_unstale":"2023-09-23T03:56:17.477158+0000","last_undegraded":"2023-09-23T03:56:17.477158+0000","last_fullsized":"2023-09-23T03:56:17.477158+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'2135","last_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_deep_scrub":"150'2135","last_deep_scrub_stamp":"2023-09-23T03:46:11.630156+0000","last_clean_scrub_stamp":"2023-09-23T03:46:11.630156+0000","objects_scrubbed":23,"log_size":2137,"log_dups_size":0,"ondisk_log_size":2137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T07:56:39.929437+0000","scrub_duration":0.23513468100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1550,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":15476,"num_read_kb":12922,"num_write":3780,"num_write_kb":3648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1470,"num_omap_keys":24,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"159'270561","reported_seq":538022,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.518509+0000","last_change":"2023-09-23T03:53:24.201856+0000","last_active":"2023-09-23T03:56:17.518509+0000","last_peered":"2023-09-23T03:56:17.518509+0000","last_clean":"2023-09-23T03:56:17.518509+0000","last_became_active":"2023-09-23T02:39:57.544670+0000","last_became_peered":"2023-09-23T02:39:57.544670+0000","last_unstale":"2023-09-23T03:56:17.518509+0000","last_undegraded":"2023-09-23T03:56:17.518509+0000","last_fullsized":"2023-09-23T03:56:17.518509+0000","mapping_epoch":18,"log_start":"156'260477","ondisk_log_start":"156'260477","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'261535","last_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_deep_scrub":"156'261535","last_deep_scrub_stamp":"2023-09-23T03:53:24.201503+0000","last_clean_scrub_stamp":"2023-09-23T03:53:24.201503+0000","objects_scrubbed":4100,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":429,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T04:11:35.049539+0000","scrub_duration":428.11806935300001,"objects_trimmed":0,"snaptrim_duration":0.012721088,"stat_sum":{"num_bytes":5261996032,"num_objects":4100,"num_object_clones":3084,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":269968,"num_read_kb":1058948,"num_write":267477,"num_write_kb":1069908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"155'1507","reported_seq":5030,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.476371+0000","last_change":"2023-09-23T03:46:12.680015+0000","last_active":"2023-09-23T03:56:17.476371+0000","last_peered":"2023-09-23T03:56:17.476371+0000","last_clean":"2023-09-23T03:56:17.476371+0000","last_became_active":"2023-09-23T02:39:49.248062+0000","last_became_peered":"2023-09-23T02:39:49.248062+0000","last_unstale":"2023-09-23T03:56:17.476371+0000","last_undegraded":"2023-09-23T03:56:17.476371+0000","last_fullsized":"2023-09-23T03:56:17.476371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'1505","last_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_deep_scrub":"150'1505","last_deep_scrub_stamp":"2023-09-23T03:46:12.679583+0000","last_clean_scrub_stamp":"2023-09-23T03:46:12.679583+0000","objects_scrubbed":26,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T09:48:01.719215+0000","scrub_duration":0.32611612400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1724,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12348,"num_read_kb":10279,"num_write":2588,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1904,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"159'251151","reported_seq":499376,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:19.488842+0000","last_change":"2023-09-23T03:56:07.582814+0000","last_active":"2023-09-23T03:56:19.488842+0000","last_peered":"2023-09-23T03:56:19.488842+0000","last_clean":"2023-09-23T03:56:19.488842+0000","last_became_active":"2023-09-23T02:39:57.544779+0000","last_became_peered":"2023-09-23T02:39:57.544779+0000","last_unstale":"2023-09-23T03:56:19.488842+0000","last_undegraded":"2023-09-23T03:56:19.488842+0000","last_fullsized":"2023-09-23T03:56:19.488842+0000","mapping_epoch":18,"log_start":"156'241060","ondisk_log_start":"156'241060","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'250590","last_scrub_stamp":"2023-09-23T03:56:07.582431+0000","last_deep_scrub":"159'250590","last_deep_scrub_stamp":"2023-09-23T03:56:07.582431+0000","last_clean_scrub_stamp":"2023-09-23T03:56:07.582431+0000","objects_scrubbed":3856,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":285,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:07:11.252103+0000","scrub_duration":284.41927812,"objects_trimmed":0,"snaptrim_duration":0.0057847840000000003,"stat_sum":{"num_bytes":4888469523,"num_objects":3856,"num_object_clones":2911,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":250605,"num_read_kb":983076,"num_write":248240,"num_write_kb":992954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"159'10521","reported_seq":18906,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-09-23T03:56:17.512846+0000","last_change":"2023-09-23T03:46:26.095424+0000","last_active":"2023-09-23T03:56:17.512846+0000","last_peered":"2023-09-23T03:56:17.512846+0000","last_clean":"2023-09-23T03:56:17.512846+0000","last_became_active":"2023-09-23T02:39:49.246084+0000","last_became_peered":"2023-09-23T02:39:49.246084+0000","last_unstale":"2023-09-23T03:56:17.512846+0000","last_undegraded":"2023-09-23T03:56:17.512846+0000","last_fullsized":"2023-09-23T03:56:17.512846+0000","mapping_epoch":13,"log_start":"45'500","ondisk_log_start":"45'500","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9614","last_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_deep_scrub":"150'9614","last_deep_scrub_stamp":"2023-09-23T03:46:26.095255+0000","last_clean_scrub_stamp":"2023-09-23T03:46:26.095255+0000","objects_scrubbed":32,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T15:45:24.169540+0000","scrub_duration":0.822139536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":21024,"num_read_kb":17573,"num_write":3101,"num_write_kb":2900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4976,"num_omap_keys":66,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"150'99","reported_seq":331,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-09-23T03:54:38.961998+0000","last_change":"2023-09-23T03:46:08.627854+0000","last_active":"2023-09-23T03:54:38.961998+0000","last_peered":"2023-09-23T03:54:38.961998+0000","last_clean":"2023-09-23T03:54:38.961998+0000","last_became_active":"2023-09-23T02:39:44.792923+0000","last_became_peered":"2023-09-23T02:39:44.792923+0000","last_unstale":"2023-09-23T03:54:38.961998+0000","last_undegraded":"2023-09-23T03:54:38.961998+0000","last_fullsized":"2023-09-23T03:54:38.961998+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'99","last_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_deep_scrub":"150'99","last_deep_scrub_stamp":"2023-09-23T03:46:08.627369+0000","last_clean_scrub_stamp":"2023-09-23T03:46:08.627369+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-09-24T12:10:25.560452+0000","scrub_duration":0.23341435499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":20708962323,"num_objects":16201,"num_object_clones":12200,"num_object_copies":48603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16201,"num_whiteouts":0,"num_read":1061254,"num_read_kb":4161092,"num_write":1051237,"num_write_kb":4204942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36211134464,"data_stored":98817703936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40327,"ondisk_log_size":40327,"up":12,"acting":12,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":14305,"num_objects":225,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":153009,"num_read_kb":128259,"num_write":34169,"num_write_kb":33009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":22554,"num_omap_keys":364,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1654784,"data_stored":28610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64776,"internal_metadata":0},"log_size":46226,"ondisk_log_size":46226,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739289,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13130220,"kb_used_data":11788580,"kb_used_omap":1245,"kb_used_meta":1340386,"kb_avail":81241620,"statfs":{"total":96636764160,"available":83191418880,"internally_reserved":0,"allocated":12071505920,"data_stored":32939885383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1275364,"internal_metadata":1372555804},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:56:10 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.84399999999999997,"15min":0.92000000000000004},"min":{"1min":0.20499999999999999,"5min":0.154,"15min":0.154},"max":{"1min":4.3369999999999997,"5min":12.862,"15min":14.617000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.55700000000000005,"15min":0.57599999999999996},"min":{"1min":0.22900000000000001,"5min":0.20799999999999999,"15min":0.19500000000000001},"max":{"1min":0.78600000000000003,"5min":6.1500000000000004,"15min":13.618},"last":0.69199999999999995}]},{"osd":1,"last update":"Sat Sep 23 03:56:10 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.48699999999999999,"15min":0.58599999999999997},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":0.56399999999999995,"5min":4.5780000000000003,"15min":10.528},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.73199999999999998,"15min":0.83699999999999997},"min":{"1min":0.26500000000000001,"5min":0.25800000000000001,"15min":0.223},"max":{"1min":3.0830000000000002,"5min":6.0190000000000001,"15min":10.625},"last":0.746}]}]},{"osd":1,"up_from":8,"seq":34359739287,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13228976,"kb_used_data":11790632,"kb_used_omap":1256,"kb_used_meta":1437079,"kb_avail":81142864,"statfs":{"total":96636764160,"available":83090292736,"internally_reserved":0,"allocated":12073607168,"data_stored":32940517497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1286974,"internal_metadata":1471569090},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,2,4,3,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 23 03:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.88600000000000001,"15min":0.876},"min":{"1min":0.20599999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.5760000000000001,"5min":33.228999999999999,"15min":33.228999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.82899999999999996,"15min":0.74199999999999999},"min":{"1min":0.24199999999999999,"5min":0.20899999999999999,"15min":0.20300000000000001},"max":{"1min":0.63400000000000001,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Sep 23 03:56:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.77800000000000002,"15min":0.74099999999999999},"min":{"1min":0.23599999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.92400000000000004,"5min":33.505000000000003,"15min":33.505000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.70599999999999996,"15min":0.751},"min":{"1min":0.23200000000000001,"5min":0.221,"15min":0.19600000000000001},"max":{"1min":0.72099999999999997,"5min":33.408999999999999,"15min":33.408999999999999},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359739285,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":13367988,"kb_used_data":11789484,"kb_used_omap":1251,"kb_used_meta":1577244,"kb_avail":81003852,"statfs":{"total":96636764160,"available":82947944448,"internally_reserved":0,"allocated":12072431616,"data_stored":32940483498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1281855,"internal_metadata":1615098049},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,10,10,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 23 03:56:12 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":0.79000000000000004,"15min":0.749},"min":{"1min":0.249,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":6.3550000000000004,"5min":7.0880000000000001,"15min":11.715},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.436,"5min":0.67300000000000004,"15min":0.83599999999999997},"min":{"1min":0.17100000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.0489999999999999,"5min":7.0469999999999997,"15min":12.058},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Sep 23 03:56:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.73499999999999999,"15min":0.59899999999999998},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":6.3159999999999998,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.372},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.78700000000000003,"15min":0.63500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23300000000000001},"max":{"1min":8.8960000000000008,"5min":8.8960000000000008,"15min":8.8960000000000008},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":8978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22543,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":14305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28195,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":5327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14038,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12069732352,"data_stored":32939171840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12071350272,"data_stored":32939352064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12070051840,"data_stored":32939180032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-23T03:56:22.089 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-23T03:56:22.907 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-23T03:56:22.907 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-23T03:56:22.907 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-23T03:56:22.908 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T03:56:26.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-09-23T03:56:29.010 INFO:tasks.ceph.osd.0:Stopped 2023-09-23T03:56:29.010 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-23T03:56:29.010 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T03:56:32.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-09-23T03:56:32.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-09-23T03:56:35.111 INFO:tasks.ceph.osd.1:Stopped 2023-09-23T03:56:35.112 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-23T03:56:35.112 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T03:56:37.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-23T03:56:37.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-09-23T03:56:41.214 INFO:tasks.ceph.osd.2:Stopped 2023-09-23T03:56:41.214 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-23T03:56:41.215 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-23T03:56:41.215 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T03:56:42.344 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-09-23T03:56:42.345 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-23T03:56:47.316 INFO:tasks.ceph.mgr.x:Stopped 2023-09-23T03:56:47.316 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-23T03:56:47.317 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-23T03:56:47.317 INFO:teuthology.orchestra.run:waiting for 300 2023-09-23T03:56:47.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-09-23T03:56:52.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-23T03:56:53.419 INFO:tasks.ceph.mon.a:Stopped 2023-09-23T03:56:53.419 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-23T03:56:53.419 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 2023-09-23T03:56:53.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T03:56:53.481 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-23T03:56:53.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T03:56:53.931 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-23T03:56:54.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T03:56:54.338 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-23T03:56:54.686 INFO:tasks.ceph:Archiving mon data... 2023-09-23T03:56:54.688 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437/data/mon.a.tgz 2023-09-23T03:56:54.688 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-09-23T03:56:54.707 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.dzT9h2e7m9 2023-09-23T03:56:54.708 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dzT9h2e7m9 2023-09-23T03:56:55.014 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.dzT9h2e7m9 2023-09-23T03:56:55.087 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.dzT9h2e7m9 is 630KB 2023-09-23T03:56:55.155 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.dzT9h2e7m9 2023-09-23T03:56:55.160 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-23T03:56:55.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-23T03:56:55.250 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-23T03:56:55.251 DEBUG:teuthology.orchestra.run.smithi002:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-23T03:56:55.311 INFO:teuthology.orchestra.run.smithi002.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-23T03:56:55.320 INFO:tasks.ceph:Archiving crash dumps... 2023-09-23T03:56:55.325 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437/remote/smithi002/crash 2023-09-23T03:56:55.326 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-23T03:56:55.381 INFO:tasks.ceph:Compressing logs... 2023-09-23T03:56:55.381 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-23T03:56:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49515.log 2023-09-23T03:56:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44622.log 2023-09-23T03:56:55.452 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45222.log 2023-09-23T03:56:55.452 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49515.log.gz 2023-09-23T03:56:55.452 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48348.log 2023-09-23T03:56:55.452 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44622.log.gz 2023-09-23T03:56:55.452 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45633.log 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45222.log.gz 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51517.log 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48348.log.gz 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53118.log 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45633.log.gz 2023-09-23T03:56:55.453 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2023-09-23T03:56:55.454 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51517.log.gz 2023-09-23T03:56:55.460 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45775.log 2023-09-23T03:56:55.461 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53118.log.gz 2023-09-23T03:56:55.461 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50297.log 2023-09-23T03:56:55.461 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2023-09-23T03:56:55.461 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45775.log.gz 2023-09-23T03:56:55.461 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47499.log 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53698.log 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50297.log.gz 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47499.log.gz 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45904.log 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2023-09-23T03:56:55.462 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53698.log.gz 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45904.log.gz 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41899.log 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45192.log 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41899.log.gz 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52515.log 2023-09-23T03:56:55.463 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53066.log 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45192.log.gz 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48186.log 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52515.log.gz 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53066.log.gz 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53248.log 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2023-09-23T03:56:55.464 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48186.log.gz 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53248.log.gz 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52865.log 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51789.log 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gz 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52865.log.gz 2023-09-23T03:56:55.465 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48618.log 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53016.log 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51789.log.gz 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48618.log.gz 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46837.log 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48373.log 2023-09-23T03:56:55.466 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53016.log.gz 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46837.log.gz 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48373.log.gz 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50572.log 2023-09-23T03:56:55.467 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2023-09-23T03:56:55.503 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50060.log 2023-09-23T03:56:55.503 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2023-09-23T03:56:55.503 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53274.log 2023-09-23T03:56:55.503 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50572.log.gz 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43839.log 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50060.log.gz 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42143.log 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53274.log.gz 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43839.log.gz 2023-09-23T03:56:55.504 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43344.log 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37952.log 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43344.log.gz 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2023-09-23T03:56:55.505 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48919.log 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50117.log 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48919.log.gz 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47407.log 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51301.log 2023-09-23T03:56:55.506 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50117.log.gz 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54309.log 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47407.log.gz 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49793.log 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51301.log.gz 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54309.log.gz 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44947.log 2023-09-23T03:56:55.507 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49793.log.gz 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49594.log 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44947.log.gz 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45069.log 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49594.log.gz 2023-09-23T03:56:55.508 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50663.log 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45069.log: -- /var/log/ceph/ceph-client.admin.38800.log 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45069.log.gz 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50663.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37952.log.gz 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50663.log.gz 2023-09-23T03:56:55.509 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53885.log 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45880.log 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.53885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53885.log.gz 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45715.log 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48522.log 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45880.log.gz 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47931.log 2023-09-23T03:56:55.510 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45715.log.gz 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51118.log 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48522.log.gz 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46024.log 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47931.log.gz 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51118.log.gz 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39849.log 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39110.log 2023-09-23T03:56:55.511 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46024.log.gz 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39849.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50510.log 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39849.log.gz 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39110.log.gz 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52029.log 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50510.log.gz 2023-09-23T03:56:55.512 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49075.log 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43531.log 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52029.log.gz 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49075.log.gz 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52210.log 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43531.log.gz 2023-09-23T03:56:55.513 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49904.log 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52210.log.gz 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49904.log.gz 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2023-09-23T03:56:55.514 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54206.log 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44273.log 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49294.log 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54206.log.gz 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47253.log 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2023-09-23T03:56:55.515 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49294.log.gz 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48979.log 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47253.log.gz 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34303.log 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53431.log 2023-09-23T03:56:55.516 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48979.log.gz 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52847.log 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34303.log.gz 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45408.log 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53431.log.gz 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52847.log.gz 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47625.log 2023-09-23T03:56:55.517 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45408.log.gz 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37815.log 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47625.log.gz 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42358.log 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37815.log.gz 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49222.log 2023-09-23T03:56:55.518 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42358.log.gz 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43066.log 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53196.log 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49222.log.gz 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38985.log 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43066.log.gz 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53196.log.gz 2023-09-23T03:56:55.519 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49023.log 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38985.log.gz 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49023.log.gz 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47849.log 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2023-09-23T03:56:55.520 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47849.log.gz 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46732.log 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38783.log 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2023-09-23T03:56:55.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46732.log.gz 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51270.log 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38783.log.gz 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51270.log.gz 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45852.log 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37651.log 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2023-09-23T03:56:55.522 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45852.log.gz 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45605.log 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37712.log 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37651.log.gz 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45605.log.gz 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39721.log 2023-09-23T03:56:55.523 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37712.log.gz 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45994.log 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45561.log 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39721.log.gz 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45994.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45994.log.gz -5 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43717.log 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47293.log 2023-09-23T03:56:55.524 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45561.log.gz 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43717.log.gz 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42492.log 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47293.log.gz 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52089.log 2023-09-23T03:56:55.525 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42492.log.gz 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52089.log.gz 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49406.log 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45435.log 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49406.log.gz 2023-09-23T03:56:55.526 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53459.log 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45435.log.gz 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39579.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz /var/log/ceph/ceph-client.admin.41509.log 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44133.log 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53459.log.gz 2023-09-23T03:56:55.527 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44824.log 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44133.log.gz 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44824.log.gz 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53589.log 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2023-09-23T03:56:55.528 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47777.log 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53589.log.gz 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40474.log 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47777.log.gz 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48949.log 2023-09-23T03:56:55.529 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40474.log.gz 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38180.log 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48949.log.gz 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38180.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38180.log.gz 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43501.log 2023-09-23T03:56:55.530 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34161.log 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37953.log 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43501.log.gz 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34161.log.gz 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50723.log 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47573.log 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37953.log.gz 2023-09-23T03:56:55.531 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43283.log 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50723.log.gz 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47573.log.gz 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43283.log.gz 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39309.log 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45807.log 2023-09-23T03:56:55.532 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2023-09-23T03:56:55.533 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39309.log.gz 2023-09-23T03:56:55.533 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48766.log 2023-09-23T03:56:55.533 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45807.log.gz 2023-09-23T03:56:55.533 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2023-09-23T03:56:55.533 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50995.log 2023-09-23T03:56:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47189.log 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48766.log.gz 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50995.log.gz 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39073.log 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38749.log 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47189.log.gz 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49318.log 2023-09-23T03:56:55.649 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39073.log.gz 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38749.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38749.log.gz 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52179.log 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43969.log 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49318.log.gz 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52179.log.gz 2023-09-23T03:56:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48862.log 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.43969.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48077.log 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43969.log.gz 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48862.log.gz 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42683.log 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48077.log.gz 2023-09-23T03:56:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52425.log 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42683.log.gz 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43942.log 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52425.log: /var/log/ceph/ceph-client.admin.43942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52425.log.gz 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43942.log.gz 2023-09-23T03:56:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44161.log 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52665.log 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50327.log 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44161.log.gz 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52665.log.gz 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51879.log 2023-09-23T03:56:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41626.log 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50327.log.gz 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51879.log.gz 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41626.log.gz 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2023-09-23T03:56:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38030.log 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42639.log 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38030.log.gz 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48838.log 2023-09-23T03:56:55.655 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51969.log 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45496.log 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48838.log.gz 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51969.log.gz 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38857.log 2023-09-23T03:56:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53485.log 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45496.log.gz 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34255.log 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38857.log.gz 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53485.log.gz 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46889.log 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51819.log 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34255.log.gz 2023-09-23T03:56:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42609.log 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46889.log.gz 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52591.log 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51819.log.gz 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53405.log 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42609.log.gz 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42882.log 2023-09-23T03:56:55.658 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52591.log.gz 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53405.log: /var/log/ceph/ceph-client.admin.51423.log 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53405.log.gz 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42882.log.gz 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52273.log 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51423.log.gz 2023-09-23T03:56:55.659 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39596.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz -5 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44762.log 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40797.log 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52273.log.gz 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39758.log 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44762.log.gz 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45934.log 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40797.log.gz 2023-09-23T03:56:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42978.log 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39758.log.gz 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45934.log.gz 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42978.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.46455.log 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42978.log.gz 2023-09-23T03:56:55.661 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40033.log 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51331.log 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46455.log.gz 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52304.log 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40033.log.gz 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49625.log 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51331.log.gz 2023-09-23T03:56:55.662 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38931.log 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52304.log.gz 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52883.log 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49625.log.gz 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49345.log 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38931.log.gz 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46641.log 2023-09-23T03:56:55.663 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52883.log.gz 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47717.log 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49345.log.gz 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46641.log.gz 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50875.log 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47717.log.gz 2023-09-23T03:56:55.664 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50875.log.gz 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43313.log 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2023-09-23T03:56:55.665 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49051.log 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43313.log.gz 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46398.log 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53672.log 2023-09-23T03:56:55.666 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49051.log.gz 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46485.log 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46398.log.gz 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39093.log 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53672.log.gz 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46054.log 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46485.log.gz 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51607.log 2023-09-23T03:56:55.667 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39093.log.gz 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45745.log 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46054.log.gz 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51607.log.gz 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39775.log 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45745.log.gz 2023-09-23T03:56:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50965.log 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54149.log 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47356.log 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50965.log.gz 2023-09-23T03:56:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45685.log 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54149.log.gz 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50418.log 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47356.log.gz 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49719.log 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45685.log.gz 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41208.log 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50418.log.gz 2023-09-23T03:56:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44548.log 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49719.log.gz 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49436.log 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41208.log.gz 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.44548.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44548.log.gz 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49436.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49436.log.gz 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52573.log 2023-09-23T03:56:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51849.log 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45964.log 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52573.log.gz 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48246.log 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51849.log.gz 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2023-09-23T03:56:55.672 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45964.log.gz 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48246.log.gz 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34406.log 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46240.log 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2023-09-23T03:56:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48022.log 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34406.log.gz 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46583.log 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46240.log.gz 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50753.log 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48022.log.gz 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52365.log 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46583.log.gz 2023-09-23T03:56:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50813.log 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50753.log.gz 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54017.log 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52365.log.gz 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46126.log 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50813.log.gz 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54043.log 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54017.log.gz 2023-09-23T03:56:55.675 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50693.log 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46126.log.gz 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40303.log 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54043.log.gz 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50693.log.gz 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2023-09-23T03:56:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40303.log.gz 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52647.log 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41011.log 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52647.log.gz 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44674.log 2023-09-23T03:56:55.677 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41011.log.gz 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43388.log 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48401.log 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44674.log.gz 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43388.log.gz 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2023-09-23T03:56:55.678 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48401.log.gz 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42735.log 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2023-09-23T03:56:55.679 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42735.log.gz 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48736.log 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53300.log 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2023-09-23T03:56:55.680 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48736.log.gz 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53913.log 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53300.log.gz 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gz 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43416.log 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53913.log.gz 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49874.log 2023-09-23T03:56:55.681 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43416.log.gz 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48276.log 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49874.log.gz 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45526.log 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48276.log.gz 2023-09-23T03:56:55.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48320.log 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45526.log: /var/log/ceph/ceph-client.admin.46153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53092.log 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38840.log 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48320.log.gz 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45282.log 2023-09-23T03:56:55.683 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53092.log.gz 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38840.log.gz 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38205.log 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45282.log.gz 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47901.log 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38205.log.gz 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38155.log 2023-09-23T03:56:55.684 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47901.log.gz 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38155.log.gz 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43467.log 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41700.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-09-23T03:56:55.685 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39451.log 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43467.log.gz 28.5% 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45526.log.gz 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52545.log 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39451.log.gz 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45252.log 2023-09-23T03:56:55.686 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52545.log.gz 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50633.log 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45252.log.gz 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2023-09-23T03:56:55.687 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50633.log.gz 2023-09-23T03:56:55.688 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2023-09-23T03:56:55.688 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2023-09-23T03:56:55.688 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44487.log 2023-09-23T03:56:55.688 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2023-09-23T03:56:55.688 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54096.log 2023-09-23T03:56:55.689 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2023-09-23T03:56:55.689 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2023-09-23T03:56:55.689 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44487.log.gz 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53753.log 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54096.log.gz 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53353.log 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53753.log.gz 2023-09-23T03:56:55.690 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49844.log 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53353.log.gz 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52829.log 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51363.log 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49844.log.gz 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51148.log 2023-09-23T03:56:55.691 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52829.log.gz 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51363.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.51363.log.gz --verbose 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.43591.log 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51148.log.gz 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46302.log 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43591.log.gz 2023-09-23T03:56:55.692 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51667.log 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46302.log.gz 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48432.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52980.log 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48645.log 2023-09-23T03:56:55.693 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51667.log.gz 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52980.log.gz 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42852.log 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48645.log.gz 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2023-09-23T03:56:55.694 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42852.log.gz 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50602.log 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39829.log: -- /var/log/ceph/ceph-client.admin.53379.log 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50207.log 2023-09-23T03:56:55.695 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50602.log.gz 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53379.log.gz 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42052.log 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38965.log 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42052.log.gz 2023-09-23T03:56:55.696 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38820.log 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46672.log 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38965.log.gz 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48566.log 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38820.log.gz 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41839.log 2023-09-23T03:56:55.697 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46672.log.gz 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48492.log 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48566.log.gz 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47108.log 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41839.log.gz 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44243.log 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48492.log.gz 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50008.log 2023-09-23T03:56:55.698 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47108.log.gz 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51240.log 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44243.log.gz 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50008.log.gz 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46702.log 2023-09-23T03:56:55.699 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51240.log.gz 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49964.log 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46702.log.gz 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48216.log 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49964.log.gz 2023-09-23T03:56:55.700 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48216.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48216.log.gz 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43193.log 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53222.log 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2023-09-23T03:56:55.701 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43193.log.gz 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45384.log 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53222.log.gz 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49934.log 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45384.log.gz 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45466.log 2023-09-23T03:56:55.702 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49934.log.gz 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37945.log 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37763.log 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45466.log.gz 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37945.log.gz 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log 2023-09-23T03:56:55.703 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37763.log.gz 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52960.log 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44457.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43561.log 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52960.log.gz 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43440.log 2023-09-23T03:56:55.704 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44406.log 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43561.log.gz 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47747.log 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43440.log.gz 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44406.log.gz 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39272.log 2023-09-23T03:56:55.705 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47747.log.gz 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51178.log 0.0% 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39272.log.gz 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37873.log 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2023-09-23T03:56:55.706 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51178.log.gz 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42549.log 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37873.log.gz 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42549.log.gz 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46611.log 2023-09-23T03:56:55.707 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52240.log 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52455.log 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46611.log.gz 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2023-09-23T03:56:55.708 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52240.log.gz 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52455.log.gz 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39218.log 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42468.log 2023-09-23T03:56:55.709 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39218.log.gz 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42468.log.gz 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53144.log 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52683.log 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51637.log 2023-09-23T03:56:55.710 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53144.log.gz 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52683.log.gz 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51637.log.gz 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz 2023-09-23T03:56:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46098.log 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50267.log 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46098.log.gz 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54069.log 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50267.log: /var/log/ceph/ceph.tmp-client.admin.30570.log 2023-09-23T03:56:55.712 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50267.log.gz 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54069.log.gz 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.tmp-client.admin.30570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30570.log.gz 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38256.log 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42792.log 2023-09-23T03:56:55.713 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50480.log 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49685.log 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42792.log.gz 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.50480.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53857.log 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50480.log.gz 2023-09-23T03:56:55.714 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48675.log 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49685.log.gz 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43664.log 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53857.log.gz 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46515.log 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48675.log.gz 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43664.log.gz 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50237.log 2023-09-23T03:56:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42267.log 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46515.log.gz 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50357.log 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50237.log.gz 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42267.log.gz 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42519.log 2023-09-23T03:56:55.716 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50357.log.gz 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53170.log 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42519.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42519.log.gz -5 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46346.log 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39039.log 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53170.log.gz 2023-09-23T03:56:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.46346.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52059.log 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46346.log.gz 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48810.log 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39039.log.gz 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38877.log/var/log/ceph/ceph-client.admin.52059.log: 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52059.log.gz 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39164.log 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48810.log.gz 2023-09-23T03:56:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38877.log.gz 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54123.log 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39164.log.gz 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54123.log.gz 2023-09-23T03:56:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50541.log 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52775.log 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gz 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50541.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50541.log.gz -5 2023-09-23T03:56:55.720 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49266.log 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47437.log 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52775.log.gz 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43253.log 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49266.log.gz 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47437.log.gz 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.43253.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45038.log 2023-09-23T03:56:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45038.log.gz 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52703.log 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51025.log 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2023-09-23T03:56:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52703.log.gz 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38948.log 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51025.log.gz 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39704.log 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52901.log 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38948.log.gz 2023-09-23T03:56:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45356.log 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39704.log.gz 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52901.log.gz 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52811.log 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45356.log.gz 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46425.log 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46210.log 2023-09-23T03:56:55.724 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52811.log.gz 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51057.log 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46425.log.gz 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.43253.log.gz 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46210.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46210.log.gz --verbose 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.47991.log 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51057.log.gz 2023-09-23T03:56:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51729.log 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47991.log.gz 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47961.log 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51088.log 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51729.log.gz 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43688.log 2023-09-23T03:56:55.726 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47961.log.gz 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51939.log 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51088.log.gz 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38894.log 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43688.log.gz 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50147.log 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51939.log.gz 2023-09-23T03:56:55.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53726.log 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38894.log.gz 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50147.log.gz 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53726.log.gz 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2023-09-23T03:56:55.728 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50448.log 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47601.log 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53991.log 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50448.log.gz 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45658.log 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47601.log.gz 2023-09-23T03:56:55.729 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42440.log 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53991.log.gz 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39505.log 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45658.log.gz 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46946.log 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42440.log.gz 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2023-09-23T03:56:55.730 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39505.log.gz 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47132.log 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46946.log.gz 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49567.log 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47132.log.gz 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46976.log 2023-09-23T03:56:55.731 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40076.log 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49567.log.gz 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48129.log 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46976.log.gz 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47821.log 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40076.log.gz 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50087.log 2023-09-23T03:56:55.732 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48129.log.gz 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49468.log 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47821.log.gz 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.50087.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51209.log 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50087.log.gz 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49468.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49468.log.gz /var/log/ceph/ceph-client.admin.46762.log 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51209.log.gz 2023-09-23T03:56:55.733 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54285.log 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46762.log.gz 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46180.log 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54285.log.gz 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48594.log 2023-09-23T03:56:55.734 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47687.log 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46180.log.gz 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.48594.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44868.log 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48594.log.gz 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49132.log 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47687.log.gz 2023-09-23T03:56:55.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44868.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44868.log.gz 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39127.log 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39002.log 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49132.log.gz 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52395.log 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39127.log.gz 2023-09-23T03:56:55.737 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38911.log 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39002.log.gz 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52395.log.gz 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52485.log 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38911.log.gz 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43097.log 2023-09-23T03:56:55.738 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49655.log 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52485.log.gz 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43097.log: -- /var/log/ceph/ceph-client.admin.53326.log 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43097.log.gz 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42711.log 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49655.log.gz 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46916.log 2023-09-23T03:56:55.739 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53326.log.gz 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51393.log 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42711.log.gz 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52739.log 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46916.log.gz 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41943.log 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2023-09-23T03:56:55.740 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.52739.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52739.log.gz 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43636.log 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41943.log.gz 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48105.log 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44333.log 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43636.log.gz 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2023-09-23T03:56:55.741 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48105.log.gz 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51759.log 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44333.log.gz 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54232.log 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44977.log 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51759.log.gz 2023-09-23T03:56:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42579.log 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54232.log.gz 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53965.log 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44977.log.gz 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44731.log 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42579.log.gz 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39019.log 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53965.log.gz 2023-09-23T03:56:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45008.log 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44731.log.gz 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.39019.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53040.log 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39019.log.gz 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.45008.log: --verbose -- /var/log/ceph/ceph-client.admin.46793.log 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45008.log.gz 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53040.log.gz 2023-09-23T03:56:55.744 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47006.log 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46793.log.gz 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39434.log 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47036.log 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47006.log.gz 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44920.log 2023-09-23T03:56:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39434.log.gz 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42762.log 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47036.log.gz 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48889.log 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44920.log.gz 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43005.log 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42762.log.gz 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48889.log: -- /var/log/ceph/ceph-client.admin.40346.log 2023-09-23T03:56:55.746 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48889.log.gz 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34338.log 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43005.log.gz 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39181.log 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40346.log.gz 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53645.log 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34338.log.gz 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42113.log 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39181.log.gz 2023-09-23T03:56:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53645.log.gz 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42113.log.gz 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39417.log 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/ceph-client.admin.41260.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2023-09-23T03:56:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51577.log 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39417.log.gz 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48706.log 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51577.log.gz 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2023-09-23T03:56:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48706.log.gz 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39363.log: /var/log/ceph/ceph-client.admin.41236.log 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53805.log 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40551.log 2023-09-23T03:56:55.750 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53805.log.gz 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40551.log: /var/log/ceph/ceph-client.admin.50388.log 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40551.log.gz 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2023-09-23T03:56:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50388.log.gz 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34279.log 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43169.log 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.34279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34279.log.gz 2023-09-23T03:56:55.752 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52611.log 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52629.log 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43169.log.gz 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43999.log 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52611.log.gz 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49192.log 2023-09-23T03:56:55.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52629.log.gz 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49817.log 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43999.log.gz 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47656.log 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49192.log.gz 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40745.log 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49817.log.gz 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2023-09-23T03:56:55.754 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47656.log.gz 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38232.log 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40745.log.gz 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50845.log 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.38232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38232.log.gz 2023-09-23T03:56:55.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41869.log 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51457.log 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52757.log 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41869.log.gz 2023-09-23T03:56:55.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43808.log 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51457.log.gz 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45312.log 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52757.log.gz 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44185.log 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43808.log.gz 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45312.log.gz 2023-09-23T03:56:55.757 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39326.log 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44185.log.gz 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39326.log.gz 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49765.log 2023-09-23T03:56:55.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42926.log 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47529.log 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49765.log.gz 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43036.log 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42926.log.gz 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-09-23T03:56:55.759 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47529.log.gz 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43036.log.gz 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50935.log 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51999.log 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47159.log 2023-09-23T03:56:55.760 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50935.log.gz 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52721.log 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51999.log.gz 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44578.log 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47159.log.gz 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52721.log.gz 2023-09-23T03:56:55.761 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44578.log: -- /var/log/ceph/ceph-client.admin.44089.log 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44578.log.gz 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43883.log 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39147.log 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43883.log.gz 2023-09-23T03:56:55.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53563.log 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.39147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39147.log.gz 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52334.log 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46865.log 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53563.log.gz 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42215.log 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52334.log.gz 2023-09-23T03:56:55.763 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46865.log.gz 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44518.log 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42215.log.gz 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.37841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47380.log 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50177.log 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44518.log.gz 2023-09-23T03:56:55.764 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47380.log.gz 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49162.log 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50177.log.gz 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49162.log.gz 2023-09-23T03:56:55.765 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-09-23T03:56:55.766 INFO:teuthology.orchestra.run.smithi002.stderr: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-09-23T03:56:57.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-23T03:57:08.431 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-09-23T03:57:10.854 INFO:teuthology.orchestra.run.smithi002.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-09-23T04:31:14.728 INFO:teuthology.orchestra.run.smithi002.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-09-23T04:46:53.343 INFO:teuthology.orchestra.run.smithi002.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-09-23T04:47:02.485 INFO:teuthology.orchestra.run.smithi002.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-09-23T04:47:02.487 INFO:teuthology.orchestra.run.smithi002.stderr: 2023-09-23T04:47:02.487 INFO:teuthology.orchestra.run.smithi002.stderr:real 50m7.062s 2023-09-23T04:47:02.487 INFO:teuthology.orchestra.run.smithi002.stderr:user 49m38.298s 2023-09-23T04:47:02.487 INFO:teuthology.orchestra.run.smithi002.stderr:sys 2m30.543s 2023-09-23T04:47:02.487 INFO:tasks.ceph:Archiving logs... 2023-09-23T04:47:02.489 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437/remote/smithi002/log 2023-09-23T04:47:02.490 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-23T04:49:20.413 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-23T04:49:20.429 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-23T04:49:20.430 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-23T04:49:20.498 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-23T04:49:20.507 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-23T04:49:20.507 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-23T04:49:20.629 INFO:teuthology.orchestra.run.smithi002.stdout: remote refid st t when poll reach delay offset jitter 2023-09-23T04:49:20.629 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================== 2023-09-23T04:49:20.629 INFO:teuthology.orchestra.run.smithi002.stdout:*hv01.front.sepi 44.4.53.2 2 u 235 256 377 0.198 -3.911 1.326 2023-09-23T04:49:20.629 INFO:teuthology.orchestra.run.smithi002.stdout:+hv02.front.sepi 63.231.80.2 3 u 1 256 377 0.190 -4.268 1.570 2023-09-23T04:49:20.630 INFO:teuthology.orchestra.run.smithi002.stdout:+hv03.front.sepi 74.6.168.72 3 u 228 256 377 0.177 -5.994 1.737 2023-09-23T04:49:20.630 INFO:teuthology.orchestra.run.smithi002.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-09-23T04:49:20.631 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-23T04:49:20.639 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-23T04:49:20.640 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-23T04:49:20.667 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-23T04:49:20.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-23T04:49:20.688 INFO:teuthology.task.internal:Duration was 8361.774446 seconds 2023-09-23T04:49:20.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-23T04:49:20.698 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-23T04:49:20.699 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-23T04:49:20.978 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-23T04:49:20.978 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2023-09-23T04:49:20.978 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-23T04:49:21.037 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-23T04:49:21.037 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-23T04:49:21.101 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-23T04:49:21.102 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-23T04:49:21.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-23T04:49:21.416 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-23T04:49:21.417 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-23T04:49:21.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-23T04:49:21.494 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-23T04:49:21.526 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2023-09-23T04:49:21.557 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-23T04:49:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T04:49:21.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-23T04:49:21.616 INFO:teuthology.task.internal:Transferring archived files... 2023-09-23T04:49:21.617 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437/remote/smithi002 2023-09-23T04:49:21.618 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-23T04:49:21.674 INFO:teuthology.task.internal:Removing archive directory... 2023-09-23T04:49:21.674 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-23T04:49:21.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-23T04:49:21.729 INFO:teuthology.task.internal:Not uploading archives. 2023-09-23T04:49:21.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-23T04:49:21.742 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-23T04:49:21.743 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-23T04:49:21.787 INFO:teuthology.orchestra.run.smithi002.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 23 04:49 /home/ubuntu/cephtest 2023-09-23T04:49:21.787 INFO:teuthology.orchestra.run.smithi002.stdout: 673422 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 23 02:40 /home/ubuntu/cephtest/mnt.0 2023-09-23T04:49:21.788 INFO:teuthology.orchestra.run.smithi002.stdout: 673849 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 23 02:41 /home/ubuntu/cephtest/mnt.0/client.0 2023-09-23T04:49:21.788 INFO:teuthology.orchestra.run.smithi002.stdout: 687732 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 23 02:41 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-23T04:49:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-23T04:49:21.789 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-09-23T04:49:21.789 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-23T04:49:21.797 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-23T04:49:21.808 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-23T04:49:21.823 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi002 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-09-23T04:49:21.823 INFO:teuthology.nuke:Checking targets against current locks 2023-09-23T04:49:21.852 DEBUG:teuthology.nuke:shortname: smithi002 2023-09-23T04:49:21.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-23T04:49:21.877 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-09-23_02:20:47-rbd:cli-wip-62891-distro-default-smithi/7401437', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-23 02:21:50.815094', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDTHhvD+lHCTnuCMxkFMRGgP2c+8SiXjzcY+NkKI2aVnWsEVb0yz3qCxaMsEyq5Or+5R29zNh9dNSgt0PSx0Ozc='} 2023-09-23T04:49:21.923 INFO:teuthology.orchestra.console.smithi002:Power off 2023-09-23T04:49:21.923 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-23T04:49:21.946 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-09-23T04:49:21.947 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T04:49:21.969 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-09-23T04:49:25.971 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T04:49:26.080 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-09-23T04:49:30.083 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-23T04:49:30.192 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-09-23T04:49:30.293 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-09-23T04:49:30.399 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/defaults msgr-failures/few objectstore/bluestore-bitmap pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8361.774446249008 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi002 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee696c23a60e05ef9b01eb4ed6317dfcb4036a8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e7d75845e58c41f0816122297fc17e70 status: fail success: false 2023-09-23T04:49:30.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-23T04:49:30.493 INFO:teuthology.run:FAIL